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Convergence criterion (GCONV=lE-8) satisfied. 



Model Fit Statistics 

Criterion 

Intercept Only 

Intercept and Covariates 

AIC 

63.086 

36.114 

SC 

64.998 

39.938 

-2 Log L 

61.086 

32.114 


Testing Global Null Hypothesis: BETA=0 


Test 

Chi-Square 

DF 

Pr> ChiSq 

Likelihood Ratio 

28.9720 

l 

<.0001 

Score 

22.3501 

l 

<.0001 

Wald 

10.8989 

l 

0.0010 
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Analysis of Maximum Likelihood Estimates 

Parameter 

DF Estimate 

Standard Error 

Wald Chi-Square 

Pr> ChiSq 

Intercept 

1 3.8982 

1.3446 

8.4044 

0.0037 

Age 

1 -0.1353 

0.0410 

10.8989 

0.0010 


Odds Ratio Estimates 

Effect 

Point Estimate 

95% Wald Confidence Limits 


Age 

0.873 

0.806 

0.947 



Association of Predicted Probabilities and Observed Responses 


Percent Concordant 

92.0 

Somers’ D 

0.851 

Percent Discordant 

6.9 

Gamma 

0.861 

Percent Tied 

1.1 

Tau-a 

0.365 

Pairs 

525 

C 

0.926 


SAS Logistic Regression Output Explanation 

The output has several measures and tables. The following sections explain. 

Output Part 1: Response Variable Summary 


Table 11-4 shows the LOGISTIC procedure. 

Table 11-4. Model Information, Output of PROC LOGISTIC 


Model Information 

Data Set 

WORK.ICE_CREAM_SALES 

Response Variable 

buy_ind 

Number of Response Levels 

2 

Model 

binary logit 

Optimization Technique 

Fisher’s scoring 


Here is an explanation of Table 11-4: 

• Data set: This is the data source. 

• Response variable: This is the dependent variable or the Y variable, the buying 
indicator in this example. 
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• Number of response levels: This is the number of levels in the dependent variable 
(mostly Yes/No); it's 1 or 0 in this example. 

• Model: This is the binary logistic regression; it’s the same as binary logit. 

• Optimization technique: Which optimization technique is used to find the regression 
coefficients? SAS chooses the most appropriate technique. 

Now consider more tables from the output in Table 11-5. 

Table 11 -5. Output Tables ofPROC LOGISTIC 

Number of Observations Read 50 

Number of Observations Used 50 


Response Profile 

Ordered Value buyjnd Total Frequency 

1 0 15 

2 1 35 

Probability modeled is buy ind='0'. 


Here is an explanation of Table 11-5: 

• Number of observations read: This is the count of observations read from the data set. 

• Number of observations used: This is the number of observations used for creating 
the model. There are no missing values or default values in the data. If there are 
missing values, SAS will give a count of those missing values separately. 

• Response profile ordered values: This is how SAS ordered the values that you 
supplied. It considered the 0 category the first level and the 1 category the 
second level. 

• Probability modeled is buy_ind= 'O': The logistic regression will finally give the 
probability of Y, which can take a value of 0 or 1. In this output, SAS is informing 
you that the model is built for 0; in other words, the output probability will be given 
for the occurrence of Y being 0. By default SAS builds the model for smaller values. 

It doesn’t really matter when you have only two levels. If the probability of 0 is 

65 percent, then the probability of 1 will obviously be 35 percent. You can use the 
descending option (refer to the case study later in this chapter, where SAS code with 
this option has been used) to directly model for the higher-order value. As per the 
output, in the ice-cream sales example, the model is built for the probability of Y 
being 0, which means a buy decision. 

• Total frequency: This is the frequency of each category in a dependent variable. 
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Output Part 2: Model Fit Summary 

In output part 2, we discuss the model convergence status and the model fit statistics. Refer to Table 11-6. 

Table 11 -6. Model Fit Summary, the Output Tables ofPROC LOGISTIC 

Model Convergence Status 

Convergence criterion (GCONV=lE-8) satisfied. 


Model Fit Statistics 

Criterion 

Intercept Only 

Intercept and Covariates 

AIC 

63.086 

36.114 

SC 

64.998 

39.938 

-2 Log L 

61.086 

32.114 


Here is an explanation of the Table 11-6: 

• Model convergence status: This is related to your model’s optimization convergence 
and precision. A detailed explanation on this is beyond the scope for this book. 

• Model fit statistics: 

• AIC: Alcaike Information Criterion is a measure that is used to compare two 
models so as to pick the best one. Generally, a model with less AIC is desired. A 
stand-alone AIC value is not of much significance, so you can ignore it for now. 

• SC: Schwarz Criterion is similar to AIC, used for comparing two models. AIC 
and SC values will be less for models that have a fewer number of predictor 
variables and high accuracy in line fitting. For example, if you are comparing 
two models to find which one is better, you should go for the model that has 
minimum AIC and SBC. Generally, AIC and SC are less for the models, which 
have high accuracy with fewer predictor variables. 


Output Part 3: Test for Regression Coefficients 

In part 3, you are testing the global null hypothesis for beta = 0. Table 11-7 shows the test results for the null 
hypothesis (that the coefficients of all the independent variables are equal to zero) versus the alternative 
hypothesis (that at least one of the confidents is nonzero). In other words, you are testing whether all the 
independent (X) variables are insignificant versus at least one of them is significant. 

H 0 : A = A=A=JB 4 =A=A. = P P = 0 

vs 

H t : P 3 ^ 0 or P 2 * 0 or P 3 ^ 0 or P t ^ 0 or P 5 * 0 or P 6 * 0 or... P p ^ 0 
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Table 11-7. Testing Global Null Hypothesis, the Output Tables ofPROC LOGISTIC 

Testing Global Null Hypothesis: BETA = 0 

Test Chi-Square DF Pr>ChiSq 


Likelihood Ratio 

28.9720 

1 

<.0001 

Score 

22.3501 

1 

<.0001 

Wald 

10.8989 

1 

0.0010 


Ail these three tests (Likelihood, Score, and Wald) are testing the previous hypothesis only. If all the 
tests show that the P-value (Pr > ChiSq) is greater than 5 percent, then there is not much evidence to reject 
the null hypothesis. This means there is not even a single variable that has a significant impact (on Y). If 
in any of the tests the P-value is less than 5 percent, then there is at least one variable that has a significant 
impact on the dependent variable. Mostly all three tests show the same result. Here in this example, the 
Chi-square tests show that the null hypothesis is rejected. This means there is at least one independent 
variable whose coefficient is not equal to zero. 


Output Part 4: The Beta Coefficients and Odds Ratio 

The coefficients, the regression parameters, or the beta values of the independent variable are given in 
Table 11-8. 

Table 11 -8. Analysis of Maximum Likelihood Estimates, the Output Tables ofPROC LOGISTIC 


Analysis of Maximum Likelihood Estimates 

Parameter 

DF 

Estimate 

Standard Error 

Wald Chi-Square 

Pr > ChiSq 

Intercept 

1 

3.8982 

1.3446 

8.4044 

0.0037 

Age 

1 

-0.1353 

0.0410 

10.8989 

0.0010 


The table (Analysis of Maximum Likelihood Estimates) shows the MLE estimates for the independent 
variable coefficients, as given by the following equation: 

gPo + Pi x i 

The value of /j Q is 3.8982, and the value of p l is -0.1353. 

The following is the logistic regression line equation based on the estimates given by SAS: 

e A>+A*i 

x ) 1 + e A+A * i 

^3.8982-0.1353*Age 
~ j + ^3.8982—0.1353*Age 

You can substitute the values of age in this equation to find the probability of buying for each customer. 

The odds ratio estimates (Table 11-9) are a little different from the previous estimates. These estimates are 
used to see the exact impact of each individual variable on the odds of the positive outcome of the model. For 
instance, you are predicting the probability of buying ice cream in this example. The odds ratio estimates tell 
you what the impact of age is on the odds of buying (in other words, the chances of buying). What is the change 
in the odds when there is a unit change in the independent variable? In this model, the change in odds is 0.873 
whenever there is a unit change in the age. 
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Table 11 -9. Odds Ratio Estimates, the Output Tables ofPROC LOGISTIC 


Odds Ratio Estimates 

Effect Point Estimate 

95% Wald Confidence Limits 

Age 0.873 

0.806 0.947 


The following is the regression line for ice-cream sales data; you substitute the age to find the predicted 
probability of buying: 

3.8982-0.1353*Age 

P(y/x) = — 

3.8982—0.1353*Age 

For example, when the age is 6, the probability of buying is 95.6 percent, and the probability of not 
buying is 4.3 percent. The odds of buying over not buying are 95 percent/5 percent, that is, 21.8981. 

When the age is increased by one unit, such as increased to 7 from 6, the probability of buying is 
95.03 percent and not buying is 4.968 percent. The odds of buying over not buying are 95.03 percent/4.968 
percent, that is, 19.12698. 

When you changed the age by one unit, from 6 to 7, the odds also changed, from 21.8981 to 19.12698. 
The odds ratio is 0.8734 (odds of buying over not buying [age=7]/the odds of buying over not buying [age=6]), 
which is the same as given in the odds ratio estimate table in SAS. The odds ratio estimate gives you the 
estimate of exact change in the odds when there is a unit change in the independent variable. Table 11-10 
illustrates the same. 


Table 11-10. Change in odds with unit change age 


Age 

Predicted Probability 
of Buying(A) 

Predicted Probability 
of Not Buying(B) 

Odds of Buying over 
Not buying (A/B) 

Change in Odds with 
Unit Change Age 

6 

96% 

4% 

21.89810 

NA 

7 

95% 

5% 

19.12698 

0.873454 

8 

94% 

6% 

16.70654 

0.873454 

9 

94% 

6% 

14.59239 

0.873454 

10 

93% 

7% 

12.74578 

0.873454 

11 

92% 

8% 

11.13285 

0.873454 

12 

91% 

9% 

9.72403 

0.873454 

13 

89% 

11% 

8.49349 

0.873454 

14 

88% 

12% 

7.41867 

0.873454 

15 

87% 

13% 

6.47987 

0.873454 

16 

85% 

15% 

5.65986 

0.873454 

17 

83% 

17% 

4.94363 

0.873454 

18 

81% 

19% 

4.31803 

0.873454 

19 

79% 

21% 

3.77160 

0.873454 

20 

77% 

23% 

3.29432 

0.873454 

21 

74% 

26% 

2.87744 

0.873454 


( continued ) 
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Table 11-10. ( continued ) 


Age 

Predicted Probability 
of Buying(A) 

Predicted Probability 
of Not Buying(B) 

Odds of Buying over 
Not buying (A/B) 

Change in Odds with 
Unit Change Age 

22 

72% 

28% 

2.51331 

0.873454 

23 

69% 

31% 

2.19526 

0.873454 

24 

66% 

34% 

1.91746 

0.873454 

25 

63% 

37% 

1.67481 

0.873454 

26 

59% 

41% 

1.46287 

0.873454 

27 

56% 

44% 

1.27775 

0.873454 

28 

53% 

47% 

1.11605 

0.873454 


From Table 11-10, you can see that the odds ratio is exactly 0.8734 whenever there is a change of 1 unit 
in the independent variable, such as the age. 


Output Part 5: Validation Statistics 

Table 11 -11 is an important table because you can use it for validating the model. This gives you an idea 
about misclassification errors and the effectiveness of the model. A higher percent of concordance is always 
desired. Somers’ D, Gamma, Tau-a, and C are derived from misclassification versus classification only. 

Table 11-11. Association of Predicted Probabilities and Observed Responses, the Output Tables of 
PROC LOGISTIC 


Association of Predicted Probabilities and Observed Responses 


Percent Concordant 

92.0 

Somers’ D 

0.851 

Percent Discordant 

6.9 

Gamma 

0.861 

Percent Tied 

1.1 

Tau-a 

0.365 

Pairs 

525 

C 

0.926 


Classifying 0 as 0 or giving a higher probability to 0 when the actual dependent variable value is 0 is 
classification without error. If the model is classifying 0 as 1 and 1 as 0 or giving a higher probability to 0 
when the actual dependent value is 1, or vice versa, it is misclassification. We will explain classification and 
misclassification in detail in the "Goodness of Fit for Logistic Regression” section. 

Individual Impact of Independent Variables 

In linear regression models, you used the P-value to check whether an independent variable has a significant 
impact on a dependent variable. The beta coefficients in linear regression follow T-distribution, so you did 
a T-test to see the impact of each variable. Here in logistic regression, the beta coefficients follow the 
Chi-square distribution. So, the probability value (P-value) of the Chi-square test tells you about the impact 
of the independent variables in logistic regression models. 
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A Chi-square test in logistic regression tests the hypotheses here: 

H 0 : The independent variable has no significant impact on the dependent 
variable. 

H : The independent variable has significant impact on the dependent variable. 

You will look at the P-value of the Chi-square test to make a decision on acceptance or rejection 
of the hypothesis. 

• If the P-value of the Chi-square test is less than 5 percent, you reject the null 
hypotheses; you reject the null hypothesis that the variable has no significant impact 
on the dependent variable. This means that the variable has some significant impact; 
hence, you keep it in the model. 

• If the P-value of the Chi-square test is greater than 5 percent, then there is not 
enough evidence to reject the null hypotheses. So, you accept the null hypothesis 
that the variable has no significant impact on the dependent variable; hence, you 
drop it from the model. Dropping such insignificant variables from the model will 
have no influence on model accuracy. 

You can take a look at the Wald Chi-square value when you are comparing two independent variables 
to decide which variable has a greater impact. If Wald Chi-square value is high, then the P-value is low. 

For example, if you are comparing the impact of two independent variables such as income and number 
of dependents on the response variable, then the variable with the higher Wald Chi-square value will be 
chosen because it has a higher impact on the dependent variable (or response variable). 

Table 11-12 is the Chi-square test table from the ice-cream sales example. 


Table 11-12. Chi-square Test, the Output Tables ofPROC LOGISTIC 


Analysis of Maximum Likelihood Estimates 

Parameter 

DF 

Estimate 

Standard Error 

Wald Chi-Square 

Pr> ChiSq 

Intercept 

1 

3.8982 

1.3446 

8.4044 

0.0037 

Age 

1 

-0.1353 

0.0410 

10.8989 

0.0010 


• H 0 : Age has no significant impact on ice-cream purchase. 

• H : Age has a significant impact on ice-cream purchase 

The P-value for the Age variable is 0.0010, which is less than 0.05 (5 percent). So, you can reject the 
null hypothesis, which would mean that age has a significant impact on ice-cream purchase. Logistic 
regressiomindependent variables 

Goodness of Fit for Logistic Regression 

In this section, we will discuss the Chi-square test in detail, and concordance. 


Chi-square Test 

The Chi-square value and the associated Chi-square test is a basic measure of goodness of fit in logistic 
regression. The Chi-square test is used for testing the null hypothesis that all the independent variables’ 
regression coefficients are zero versus at least one of the independent variables is significant. The Chi-square 
test is similar to F-test in linear regression. 
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• A=A=A=A=A=A. = P P =0. 

• This is same as P 1 = 0 and P 2 = 0 and P 3 = 0 and /3 4 = 0. and P p = 0. 

• This means that the overall model is insignificant or it has no explanatory power. 

H i = 

• A 0orP 2 ^0 orP 3 ^0 orP t ^0 orP 5 ^0 or P 6 ^0 or...p p ^0. 

• This is same as at least one P * 0 . 

• This means that at least one variable is significant, or the model has some 
explanatory power. 

If the P-value of the Chi-square test is greater than 5 percent, then the model is at risk. If the P-value 
is more than 5 percent, then you have to accept the null hypothesis, which means the overall model 
is insignificant. There is no point in proceeding further, and none of the independent variables has 
significant impact on the dependent variable. It forces you to collect more data and look for more impactful 
independent variables. In the SAS output, you see three Chi-square tests—Likelihood Ratio, Score, and 
Wald—all of them for the same purpose. Generally, all these tests show the same result. 


Concordance 

As discussed earlier, the concordance measure gives you an estimate of accuracy or the goodness of fit of 
your logistic regression model. You will consider an example in this section to understand concordance and 
discordance. 

Imagine a simple data set where there are just 10 observations. Five records of the dependent 
variable take a value of 1; the other five are at 0. After building the model, you will determine the predicted 
probability of the dependent variable for each record. It looks like Table 11-13. 


Table 11-13. Predicted Probability of Dependent Variable 


DEPENDENT VARIABLE 

PREDICTED PROBABILITY 

0 

Pi 

0 

P2 

0 

P3 

0 

P4 

0 

P5 

1 

P6 

1 

P7 

1 

P8 

1 

P9 

1 

P10 
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You can make as many as 25 pairs using five Os and five is along with their corresponding probabilities. 

(0 PI - 1 P6), (0 PI - 1 P7), (0 PI - 1 P8), (0 PI - 1 P9), (0 PI - 1 P10) 

(0 P2 - 1 P6), (0 P2 - 1 P7), (0 P2 - 1 P8), (0 P2 - 1 P9), (0 P2 - 1 P10) 

(0 P3 - 1 P6), (0 P3 - 1 P7), (0 P3 - 1 P8), (0 P3 - 1 P9), (0 P3 - 1 P10) 

(0 P4 - 1 P6), (0 P4 - 1 P7), (0 P4 - 1 P8), (0 P4 - 1 P9), (0 P4 - 1 P10) 

(0 P5 - 1 P6), (0 P5 - 1 P7), (0 P5 - 1 P8), (0 P5- 1 P9), (0 P5 - 1 P10) 

Take the first pair (0 PI - 1 P6). If you built a model to predict the value 0, then PI should be higher 
than P6. A good model should give higher probabilities to zero and lower probabilities to 1. ffence, PI 
should be greater than P6 in the first pair, PI should be greater than P7 in the second pair, and P5 should be 
greater than P10 in the last pair. This accurate classification is called concordance. Percent concordance is the 
percentage of concordance cases in all possible pairs. The higher the percent concordance, the higher the 
accuracy of the model. 

If in the first pair (0 PI - 1 P6) if PI is less than P6, then it is not a good sign. You built a model to 
predict zero, and the model is finally predicting higher probabilities for 1 and lower probability for 0. This 
misclassification is called discordance. The percentage of discordant pairs in overall (all possible pairs) is 
called percent discordance. 

For example, take the data pair from the ice-cream sales data. 


Age 

Buy 

6 

0 

45 

l 


The pair is (0 PI - 1 P2). If you substitute the values of age in the model, you get pi = 95 percent, 
p2 = 0.6 percent. So, you are correct in predicting that if a person is 6 years old, he has a higher probability of 
buying the ice cream (pi = 95 percent), whereas a 45-year-old person has almost zero probability of buying 
(p2 = 0.6 percent). The predictions are actually matching with the original observations from the table. 

This is what you expect from a good model; when you observe 0 (in actual values) and still try to predict 
it, the model probability should suggest the same. Similarly, when you take 1 and try to predict it using the 
model, the models should suggest it. When you take a pair (0,1) and the probabilities are (0 PI - 1 P2), then 
at least you expect the model to give you predictions PI and P2 as PI > P2. This is concordance. The opposite 
of this is discordance. 

Tied cases are those for which our model gives the same probability of 0.5 for 0 and 0.5 for 1. 

Table 11-14 is the output for ice-cream sales example. 

Table 11-14. Association of Predicted Probabilities and Observed Responses, 
the Output Tables ofPROC LOGISTIC on Ice-Cream Sales Data 


Association of Predicted Probabilities and Observed Responses 


Percent Concordant 

92.0 

Somers’ D 

0.851 

Percent Discordant 

6.9 

Gamma 

0.861 

Percent Tied 

1.1 

Tau-a 

0.365 

Pairs 

525 

c 

0.926 
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Total pairs = 15*35 =525 (15 zeros, 35 ones) 

• Percent concordant = Percent of right classification = 92.0 

• Percent discordant = Percent of wrong classification = 6.9 

• Percent tied = 1.1 (100-(92.0+6.9)) 

The ice-cream sales model that you built is a good one for prediction since the percent concordance is 
92 percent. Any concordance greater than 70 percent is good; greater than 80 percent is ideal. 

Prediction Using Logistic Regression 

The logistic regression line can be stated by substituting the beta coefficients in the following equation: 

g A> + Pi X 1 +02 X 2 +/*3 *3 +At X 4 +. + Pp X p 

p( y / x ) — ^ ft+ftji+ftij+ft* ,+p,x,+ . +p p x p 

In the ice-cream sales example, you can manually substitute the values of age and get the probability 
of buying, that is, the probability of y=0. There is a note in the SAS output that says "Probability modeled is 
buy_ind=‘0!" This means that the probability values are for buy_ind=0, or you are getting the probability of 0 
when you substitute the value of Age. 


p(y/x) 


^ 3.8982—0.1353*Age 
\ g 3.8982-0.1353*Age 


For example, say you want to find the probability that a person who is 12 will buy the ice cream. The 
following is the substitution: 


^ 3.8982—0.1353*12 

P(y — ^ / X — 12) — ^ + ^3.8982-0.1353*12 


P(y = 0 /x = 12) = 0.907 (almost 91%) 


Multicollinearity in Logistic Regression 

If you have multiple independent variables, then, as expected, you will have multiple beta coefficients. 

A +fi*l + ftft +ftft + Pix, +. +P p x p 

P{y /X )- 1 + e p„+p 1 x 1 +p 2 x 2 +p a x 3 +p i x i +. . +P p X p 

We discussed multicollinearity in regression in Chapter 10 . It is an interdependency of the 
independent variables that upsets the model accuracy by increasing the standard deviation of the 
beta coefficients. We discussed the effects of multicollinearity and how to detect and remove it in 
linear regression. Is multicollinearity a problem in multiple logistic regression too? How different is 
multicollinearity in logistic regression? 

Figure 11-10 shows multicollinearity in linear regression. XI, X2, X3, X4, X5...Xp are independent 
variables, and Y is a dependent variable that is linearly dependent on all of these. 
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Figure 11-10. Multicollinearity in linear regression 

You need to take the note of one fact here; multicollinearity is connected with independent variables, 
and their relation with the dependent variable has nothing to do with it. Whether it is a linear regression 
or a nonlinear regression, you can have scenarios where the independent variables are interdependent 
(Figure 11-11). 



Figure 11-11. Multicollinearity in nonlinear regression 

In simple terms, multicollinearity is an issue with logistic regression as well. The dependent variable 
is nowhere on the scene while you are dealing with multicollinearity. So, it doesn't really matter whether 
you are building a linear or nonlinear regression model. Multicollinearity dents the accuracy of a logistic 
regression model as well. You have to be careful of multicollinearity while fitting a multiple logistic 
regression line. The steps are same as discussed in linear regression multicollinearity, shown here: 

1. Detect or conform multicollinearity using VIF values. If VIF is more than 5, then 
there is some multicollinearity within the independent variables. 

2. Once you find the existence of multicollinearity, you can drop one of the 
troublesome variable by looking at the individual variable’s Wald Chi-square 
value. Keep the variables that have a higher Wald Chi-square value. 
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No VIF Option in PROC LOGISTIC 

Multicollinearity has nothing to do with logistic or linear regression. This phenomenon is entirely centered 
on independent variables. Accordingly, SAS doesn’t have a separate option of VTF in logistic regression. 

You have to follow the same steps as you followed with linear regression as far as the detection of 
multicollinearity is concerned. For variable selection or dropping, you can use logistic regression. In simple 
words, PROC LOGISTIC doesn't have a VIF option; only PROC REG has that option. 

If you write the VIF option in logistic regression, then SAS gives you an error. The following is 
one such example: 

NOTE: The SAS System stopped processing this step because of errors. 

704 proc logistic data=loans_data; 

705 model SeriousDlqin2yrs = utilization Age NumJLoans Num_dependents Monthlylncome 
705! Num_Savings_Acccts DebtRatio/vif; 

22 

202 

ERROR 22-322: Syntax error, expecting one of the following: ;, ABSFCONV, AGGREGATE, ALPHA, 
BACKWARD, BEST, BINWIDTH, BUILDRULE, CL, CLODDS, CLPARM, CODING, CONVERGE, 
CORRB, COVB, CT, CTABLE, DETAILS, DSCALE, EXPB, FAST, FCONV, FIRTH, GCONV, 
HIERARCHY, INCLUDE, INFLUENCE, IPLOTS, ITPRINT, L, LACKFIT, LINK, MAXITER, 
MAXSTEP, NOCHECK, NODUIWPRINT, NOFIT, NOINT, NOLOGSCALE, OFFSET, OUTROC, 
PARMLABEL, PEVENT, PL, PLCL, PLCONV, PLRL, PPROB, PSCALE, RIDGING, RISKLIMITS, 
ROCEPS, RSQUARE, SCALE, SELECTION, SEQUENTIAL, SINGULAR, SLE, SLENTRY, SLS, 
SLSTAY, START, STB, STEPWISE, STOP, STOPRES, TECHNIQUE, WALDCL, WALDRL, XCONV. 
ERROR 202-322: The option or parameter is not recognized and will be ignored. 

706 run; 

So, in the case of logistic regression, you have to use PROC REG and follow the procedure explained in 
Chapter 10. 

Logistic Regression Final Check List 

Up to this point, we have discussed all the important concepts related to regression analysis. Logistic 
regression is just another type of regression. It is one of the most widely used nonlinear regression models 
across numerous industry verticals. The following is a check list to be used while building a logistic 
regression model: 

1. Applicability: Look at the data and the dependent variables. Is it categorical? Yes/ 

No, 0/1, Win/Loss, and so on, are the types of response variable outcomes where 
you can apply logistic regression. 

2. Chi-square value: Look at the overall Chi-square value to decide whether a model 
is significant. If the Chi-square test fails, then stop it right there; the overall model 
itself is not significant. Chi-square is not going to tell you anything about the 
precise accuracy of a model. 

3. VIF: The multicollinearity issue needs to be solved in same fashion as in linear 
regression models (using PROC REG with the VIF option). Identify and resolve 
it by dropping the troublesome variables or by using PCA or FA, as explained in 
regression analysis in Chapter 10. 
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4. Overall accuracy/concordance-. Determine the accuracy of a mode by looking 
at its concordance and C values. The higher the value, the better the model. 

If the values of concordance and C are not satisfactory, then you may think 
of collecting some more data or adding better, more impactful independent 
variables, which will improve the overall model performance. 

5. Individual impact/Wald Chi-square value-. Look at the individual impact of each 
variable by looking at the Wald Chi-square value. Drop the insignificant variables 

Loan Default Prediction Case Study 

As we have done throughout this book, we will revise the concepts learned in this chapter using a real-life 
case study. 


Background and Problem Statement 

ABC Bank has a personal loan product. It needs to be very sure about the payback capability of customers 
before it can approve a loan. The bank gets thousands of loan applications from its prospective customers, 
and it needs to have a flawless scientific method to approve or reject the loan applications. The bank decides 
to use data analytics to judge the risk associated with each customer. 

ABC Bank has tons of historical data on customer profiles, their previous loans, their credit card 
spending, their bank accounts, and so on. The bank has historic data of both creditworthy and non-credit- 
worthy customers. Identifying the non-credit-worthy applicants before approving a loan saves millions of 
dollars for the bank. So, it is important to quantify the risk associated with each customer at the time of loan 
approval. 


Objective 

Analyze the historical loans data and build a model that will help the bank segregate the customers, based 
on the associated risk, at the time of loan approval. 


Data Set 

The bank's historical data contains the records of both defaults and nondefaulters. It is collected over a period 
of two years. The data set has a record of 150,000 customers. It also has a response/dependent variable, which 
indicates the creditworthiness of each customer. Some predictor/independent variables are also there in 
each record, which indicate the customer’s financial health. The data set given here is reasonably clean for 
the model-building exercise. The data validation and sanitization have already been done. Also, there are no 
oudiers or missing values in the variables. 

Table 11-15 contains the details of the data set (the data dictionary). 
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Table 11-15. The Details of Data Set Used in the ABC Bank Example 


Variable Name 

Description 

Type 

Cust_id 

Customer ID; unique for each customer. 

Number 

SeriousDlqin2yrs 

Person experienced BK/CO. 

BK: Bankruptcy. This is when a debtor realizes that she is unable 
to pay her bills. When any individual or business is unable to pay 
their creditors, bankruptcy is a legal option that can help them 
to get relief from their due payments and debt. 

CO: A customer fails to make the minimum payment due for a 
period of six months or more. 

1 - Yes BK or CO: Bad customer. 

0 - No BK or CO: Good customer. 

1/0 

Utilization 

Average monthly utilization of credit limit. 

Percentage 

Age 

Age of the customer. 

Number 

Num loans 

Number of loans including mortgage, vehicle, and personal loans. 

Number 

Numdependents 

Number of dependents. 


Monthlylncome 

Average monthly income of the customer. 

Number 

Num Savings Acccts 

Number of savings accounts a customer has. 

Number 

DebtRatio 

Monthly debt payments, alimony, living costs divided by monthly 
gross income. 

Percentage 


Data Import 

As a first step, you will import the data and do a quick check on the number of records, variables, formats, 
and so on. 

The following is the code for importing the data set CusLomerLoans.csv: 

PROC IMPORT 0UT= WORK.loans_data 

DATAFILE= "C:\Users\VENKAT\Google Drive\Training\Books\Content\ll. 

Logistic RegressionXData Logistic Regression\Customer_Loans.csv" 

DBMS=CSV REPLACE; 

GETNAMES=YES; 

DATAR0W=2; 

RUN; 


Here are the notes from the log file for the previous code: 

NOTE: The infile 'C:\UsersWENKAT\Google Drive\Training\Books\Content\ll. Logistic 
RegressionXData Logistic Regression\Customer_Loans.csv' is: 

Filename=C: XUsersXVENKATXGoogle DriveXTrainingXBooksXContentXll. Logistic 
RegressionXData Logistic Regression\Customer_Loans.csv, 

RECFM=V,LRECL= 32767 ,File Size (bytes)=69l8l39. 

Last Modified=28Aug20l4:11:52:33, 

Create Time=28Aug20l4:ll:l2:22 
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NOTE: 150000 records were read from the infile 'C:\UsersWENKAT\Google 
Drive\Training\Books\Content\ll. Logistic RegressionXData Logistic 
RegressionXCustomerLoans.csv'. 

The minimum record length was 26. 

The maximum record length was 48. 

NOTE: The data set WORK.LOANS_DATA has 150000 observations and 9 variables. 

NOTE: DATA statement used (Total process time): 
real time 1.23 seconds 

cpu time 1.24 seconds 

150000 rows created in WORK.LOANS_DATA from C:\UsersWENKAT\Google 
Drive\Training\Books\Content\ll. Logistic RegressionXData Logistic 
Regression\Customer_Loans.csv. 

NOTE: WORK.LOANS_DATA data set was successfully created. 

NOTE: PROCEDURE IMPORT used (Total process time): 
real time 1.49 seconds 

cpu time 1.51 seconds 

The following is the code for PROC CONTENTS and print data to see the variables and the snapshot of 
the data: 

proc contents data=loans_data varnum; 

run; 

proc print data=loans_data(obs=20); 

run; 


Tables 11-16 and 11-17 list the output of these code snippets. 


Table 11-16. The Output of proc contents Procedure on loans_data Data Set 


The SAS System 

The CONTENTS Procedure 



Data Set Name 

WORK.LOANSJDATA 

Observations 

150000 

Member Type 

DATA 

Variables 

9 

Engine 

V9 

Indexes 

0 

Created 

Tuesday, July 01, 2008 

12:00:00 AM 

Observation Length 

72 

Last Modified 

Tuesday, July 01, 2008 

12:00:00 AM 

Deleted Observations 

0 

Protection 


Compressed 

NO 

Data Set Type 


Sorted 

NO 

Label 




Data Representation 

WINDOWS_32 



Encoding 

wlatinl Western (Windows) 
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Engine/Host Dependent Information 

Data Set Page Size 

8192 

Number of Data Set Pages 

1328 

First Data Page 

1 

Max Obs per Page 

113 

Obs in First Data Page 

85 

Number of Data Set Repairs 

0 

Filename 

C:\Users\VENKAT\AppData\Local\Temp\SASTemporary 

Files\_TD4892\loans_data.sas7bdat 

Release Created 

9.0201M0 

Host Created 

W32_VSPRO 


Variables in Creation Order 

# 

Variable 

Type 

Len 

Format 


Informat 

l 

Cust id 

Num 

8 

BEST12. 


BEST32. 

2 

SeriousDlqin2yrs 

Num 

8 

BEST12. 


BEST32. 

3 

Utilization 

Num 

8 

BEST12. 


BEST32. 

4 

Age 

Num 

8 

BEST12. 


BEST32. 

5 

Numloans 

Num 

8 

BEST12. 


BEST32. 

6 

Num dependents 

Num 

8 

BEST12. 


BEST32. 

7 

Monthlylncome 

Num 

8 

BEST12. 


BEST32. 

8 

Num Savings Acccts 

Num 

8 

BEST12. 


BEST32. 

9 

DebtRatio 

Num 

8 

BEST12. 


BEST32. 

Table 11-17. 

The Output ofproc print Procedure on loans_data Data Set (obs=20) 



Obs 

Custjd 

Serious utilization 
Dlqin2yrs 

Age Num_ 
loans 

Num_ 

dependents 

Monthly 

Income 

Num_ Debt 

Savings_ Ratio 

Acccts 

1 

100001 

1 0.766126609 

45 6 

2 

9120 

2 

0.852982129 

2 

100002 

0 0.957151019 

40 0 

1 

4600 

2 

0.121876201 

3 

100003 

0 0.65818014 

38 0 

0 

5042 

2 

0.085113375 

4 

100004 

0 0.233809776 

30 0 

0 

5300 

2 

0.036049682 

5 

100005 

0 0.9072394 

49 1 

0 

6357 

2 

0.024925695 

6 

100006 

0 0.213178682 

74 1 

1 

5500 

2 

0.375606969 

7 

100007 

0 0.305682465 

57 3 

0 

6357 

2 

0.274502 


(i continued ) 
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Table 11-17. ( continued ) 


Obs 

Custjd 

Serious 

Dlqin2yrs 

utilization 

Age 

Num_ 

loans 

Num_ 

dependents 

Monthly 

Income 

Num_ 

Savings. 

Acccts 

Debt 

Ratio 

8 

100008 

0 

0.754463648 

39 

0 

0 

5500 

2 

0.209940017 

9 

100009 

0 

0.116950644 

27 

0 

0 

6357 

2 

0.274502 

10 

100010 

0 

0.189169052 

57 

4 

2 

6357 

2 

0.606290901 

11 

100011 

0 

0.644225962 

30 

0 

0 

4500 

2 

0.30947621 

12 

100012 

0 

0.01879812 

51 

2 

2 

8501 

2 

0.53152876 

13 

100013 

0 

0.010351857 

46 

2 

2 

14454 

2 

0.298354075 

14 

100014 

1 

0.964672555 

40 

1 

2 

13700 

2 

0.432964747 

15 

100015 

0 

0.019656581 

76 

1 

0 

6357 

2 

0.274502 

16 

100016 

0 

0.548458062 

64 

1 

2 

13362 

2 

0.209891754 

17 

100017 

0 

0.061086118 

78 

2 

0 

6357 

2 

0.274502 

18 

100018 

0 

0.166284079 

53 

0 

0 

10800 

1 

0.18827406 

19 

100019 

0 

0.221812771 

43 

1 

2 

5280 

2 

0.527887839 

20 

100020 

0 

0.602794411 

25 

0 

0 

2333 

2 

0.065868263 


You have all the variables in the data, and it looks like all the variables are populated. The data looks 
clean. If not, you need to first explore the data, validate it for the accuracy, and finally clean it by treating 
missing values and outliers in the data. You need to use SAS procedures such as PROC CONTENTS, PROC 
FREQ, and PROC UNIVARIATE, which have been discussed at length in data exploration in Chapter 7. 


Model Building 

The response variable in the data set is binary. SeriousDlqin2yrs is the dependent variable that you are 
planning to predict. It takes two values: 

• 1, Yes BK or CO: Bad customer 

• 0, No BK or CO: Good customer 

Logistic regression is a good choice for predicting the default probability of a customer based on 
characteristics such as number of loans, debt ratio, monthly income, and so on. 

The following is the code for building a logistic regression line: 

proc logistic data=loans_data; 

model SeriousDlqin2yrs = utilization Age NumJLoans Num_dependents Monthlylncome 
Num_Savings_Acccts DebtRatio; 

run; 
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The following are the messages from the log file when the previous code was executed: 

701 proc logistic data=loans_data; 

702 model SeriousDlqin2yrs = utilization Age NumJLoans Num_dependents Monthlylncome 

702! Num_Savings_Acccts DebtRatio; 

703 run; 

NOTE: Writing HTML Body file: sashtml44.htm 

NOTE: PROC LOGISTIC is modeling the probability that SeriousDlqin2yrs='O'. One way to change 
this to model the probability that SeriousDlqin2yrs='l' is to specify the response 
variable option EVENT='l'. 

NOTE: Convergence criterion (GC0NV=lE-8) satisfied. 

NOTE: There were 150000 observations read from the data set WORK.LOANS_DATA. 

NOTE: PROCEDURE LOGISTIC used (Total process time): 
real time 4.92 seconds 

cpu time 3.58 seconds 

Table 11-18 shows the SAS output for this code. 


Table 11-18. The Output of proc logistic on loansjlala 


The SAS System 

The LOGISTIC Procedure 


Model Information 


Data Set 

WORK. LOAN S_D ATA 

Response Variable 

SeriousDlqin2yrs 

Number of Response Levels 

2 

Model 

binary logit 

Optimization Technique 

Fisher’s scoring 



Number of Observations Read 

150000 

Number of Observations Used 

150000 


Response Profile 

Ordered Value 

SeriousDlqin2yrs 

Total Frequency 

l 

0 

139974 

2 

l 

10026 


Probability modeled is SeriousDlqin2yrs=‘0! 
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Model Convergence Status 

Convergence criterion (GCONV=lE-8) satisfied. 


Model Fit Statistics 

Criterion 

Intercept Only 

Intercept and Covariates 

AIC 

73618.167 

57641.254 

SC 

73628.085 

57720.601 

-2LogL 

73616.167 

57625.254 




Testing Global Null Hypothesis: BETA=0 


Test Chi-Square DF 

Pr>ChiSq 


Likelihood Ratio 

15990.9128 

7 

<.0001 

Score 

14078.3178 

7 

<.0001 

Wald 

11514.4825 

7 

<.0001 


Analysis of Maximum Likelihood Estimates 


Parameter 

DF 

Estimate 

Standard Error 

Wald Chi-Square 

Pr> ChiSq 

Intercept 

1 

1.1431 

0.0622 

337.3241 

<.0001 

utilization 

1 

-1.9713 

0.0312 

3999.3621 

<.0001 

Age 

1 

0.0108 

0.000832 

167.2586 

<.0001 

Num loans 

1 

-0.2464 

0.0106 

538.4191 

<.0001 

Num dependents 

1 

-0.1941 

0.00960 

408.6747 

<.0001 

Monthlylncome 

1 

0.000426 

5.983E-6 

5078.5897 

<.0001 

Num Savings Acccts 

1 

-0.00945 

0.00517 

3.3408 

0.0676 

DebtRatio 

1 

-1.1702 

0.0510 

526.4205 

<.0001 
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Odds Ratio Estimates 


Effect 

Point Estimate 95% Wald Confidence Limits 


Utilization 

0.139 

0.131 

0.148 

Age 

1.011 

1.009 

1.012 

Num loans 

0.782 

0.766 

0.798 

Num dependents 

0.824 

0.808 

0.839 

Monthlylncome 

1.000 

1.000 

1.000 

Num Savings Acccts 

0.991 

0.981 

1.001 

DebtRatio 

0.310 

0.281 

0.343 


Association of Predicted Probabilities and Observed Responses 


Percent Concordant 

82.8 

Somers' D 

0.664 

Percent Discordant 

16.5 

Gamma 

0.668 

Percent Tied 

0.7 

Tau-a 

0.083 

Pairs 

1403379324 

c 

0.832 


Now we will discuss the output in detail. 

Some 150,000 customer records are read from the data set (Table 11-19) and are being used for this 
analysis. There are 139,974 good accounts and 10,026 bad accounts (Table 11-20). In the real-life bank loan 
portfolios, you would expect more than 90 percent of customers are good. 

Table 11-19. Number of Observations in loans__data 

Number of Observations Read 150000 

Number of Observations Used 150000 


Table 11 -20. Frequency ofO and 1 in SeriousDlqin2yr Variable 


Response Profile 

Ordered Value 

SeriousDlqin2yrs 

Total Frequency 

l 

0 

139974 

2 

l 

10026 


Probability modeled is SeriousDlqin2yrs='0 


You may be wondering what you are trying to predict if good and bad accounts are already given. Why 
don’t you directly use them? Remember, this is historical data, where you already know who is good and 
who is bad. You are trying to use this data to fit a model, which will predict the probability of default (by new 
customers), given independent variables such as income, loans, debits, and utilization. You can use this 
model for future loan approvals for the new customers, even if they are not in the historic data. For all future 
customers, you will solicit the data on independent variables at the time of loan application, which will be used 
to assess the risk associated with each customer. 
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Of the following tables, Table 11-21 talks about the model convergence and Table 11-22 gives 
the Chi-square test results. 

Table 11 -21. Model Convergence Status 

Model Convergence Status 

Convergence criterion (GCONV=lE-8) satisfied. 


Table 11 -22. Chi-square Test Results for the Model 

Testing Global Null Hypothesis: BETA=0 


Test Chi-Square DF Pr>ChiSq 


Likelihood Ratio 

15990.9128 

7 

<.0001 

Score 

14078.3178 

7 

<.0001 

Wald 

11514.4825 

7 

<.0001 


Looking at the Chi-square test results of the output (Table 11-22), you can confirm that the model is 
significant. All the P-values for the Chi-square tests are less than 5 percent (0.05). So, it is safe to conclude 
that the model is significant. In other words, at least one variable in the independent variables list has a 
significant impact on the response/dependent variable (SeriousDlqin2yrs). Take a look at the goodness of 
fit discussed earlier in the chapter to get a clear idea of howto interpret Table 11-22. 

Table 11 -23 is about variable coefficient estimates. The variable signs indicate that some of them 
have positive impact on dependent variable, while others impact negatively. Before looking at the sign of 
coefficients and their values or making any attempt to write the model equation, you need to make sure 
there is no interdependency within the independent variables. You are well aware that multicollinearity is a 
concern in logistic regression as well. You can’t trust these coefficients unless you make sure that there is no 
multicollinearity in the model. 


Table 11 -23. Variable Coefficient Estimates 


Analysis of Maximum Likelihood Estimates 

Parameter 

DF 

Estimate 

Standard Error 

Wald Chi-Square 

Pr> ChiSq 

Intercept 

l 

1.1431 

0.0622 

337.3241 

<.0001 

utilization 

l 

-1.9713 

0.0312 

3999.3621 

<.0001 

Age 

l 

0.0108 

0.000832 

167.2586 

<.0001 

Num loans 

l 

-0.2464 

0.0106 

538.4191 

<.0001 

Num dependents 

l 

-0.1941 

0.00960 

408.6747 

<.0001 

Monthlylncome 

l 

0.000426 

5.983E-6 

5078.5897 

<.0001 

Num Savings Acccts 

l 

-0.00945 

0.00517 

3.3408 

0.0676 

DebtRatio 

l 

-1.1702 

0.0510 

526.4205 

<.0001 
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Handling Multicollinearity 

To reiterate, multicollinearity is an issue affected by only the list of independent variables. It has nothing to 
do with the relation of the independent variables to the dependent variables. Whether you are building a 
linear or logistic regression line, the treatment of the multicollinearity issue remains the same. 

The following is the code for detecting multicollinearity using the VIF option: 

proc logistic data=loans_data; 

model SeriousDlqin2yrs = utilization Age Num_loans Num_dependents Monthlylncome 
Num_Savings_Acccts DebtRatio/vif; 

run; 


Remember that PROC LOGISTIC has no VIF option. If a VIF option is used, this PROC code throws 

an error, as shown here: 

NOTE: The SAS System stopped processing this step because of errors. 

704 proc logistic data=loans_data; 

705 model SeriousDlqin2yrs = utilization Age NumJLoans Num_dependents Monthlylncome 

705! Num_Savings_Acccts DebtRatio/vif; 

22 

202 

ERROR 22-322: Syntax error, expecting one of the following: ;, ABSFCONV, AGGREGATE, ALPHA, 
BACKWARD, BEST, BINWIDTH, BUILDRULE, CL, CLODDS, CLPARM, CODING, CONVERGE, 
CORRB, COVB, CT, CTABLE, DETAILS, DSCALE, EXPB, FAST, FCONV, FIRTH, GCONV, 
HIERARCHY, INCLUDE, INFLUENCE, IPLOTS, ITPRINT, L, LACKFIT, LINK, MAXITER, 
MAXSTEP, NOCHECK, NODUIWPRINT, NOFIT, NOINT, NOLOGSCALE, OFFSET, OUTROC, 
PARMLABEL, PEVENT, PL, PLCL, PLCONV, PLRL, PPROB, PSCALE, RIDGING, RISKLIMITS, 
ROCEPS, RSQUARE, SCALE, SELECTION, SEQUENTIAL, SINGULAR, SLE, SLENTRY, SLS, 
SLSTAY, START, STB, STEPWISE, STOP, STOPRES, TECHNIQUE, WALDCL, WALDRL, XCONV. 

ERROR 202-322: The option or parameter is not recognized and will be ignored. 

706 run; 

So, you need to use PROC REG for the treatment of multicollinearity here as well. 

proc reg data=loans_data; 

model SeriousDlqin2yrs = utilization Age NumJLoans Num_dependents Monthlylncome 

Num_Savings_Acccts DebtRatio/vif; 

run; 
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Table 11-24 shows the output for the previous code. 

Table 11 -24. Output ofproc reg on loans_data 

The SAS System 


150000 

150000 


The REG Procedure 
Model: MODEL 1 

Dependent Variable: SeriousDlqin2yrs 
Number of Observations Read 
Number of Observations Used 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr > F 

Model 

7 

87809867 

125.44267 

2219.38 

<.0001 

Error 

149992 

8477.76349 

0.05652 



Corrected Total 

149999 

9355.86216 





Root MSE 

0.23774 

R-Square 

0.0939 

Dependent Mean 

0.06684 

Adj R-Sq 

0.0938 

CoeffVar 

355.68877 




Parameter Estimates 


Variable 

DF 

Parameter 

Estimate 

Standard 

Error 

t Value 

Pr > |t| 

Variance 

Inflation 

Intercept 

l 

0.11651 

0.00342 

34.11 

<.0001 

0 

utilization 

l 

0.14700 

0.00194 

75.88 

<.0001 

1.11193 

Age 

l 

-0.00057428 

0.00004428 

-12.97 

<.0001 

1.13520 

Num loans 

l 

0.00849 

0.00067032 

12.67 

<.0001 

1.45625 

Num dependents 

l 

0.01100 

0.00058660 

18.75 

<.0001 

1.11697 

Monthlylncome 

l 

-0.00001339 

1.931776E-7 

-69.33 

<.0001 

1.22390 

Num Savings Acccts 

l 

0.00050943 

0.00028330 

1.80 

0.0722 

1.00006 

DebtRatio 

l 

0.06312 

0.00356 

17.74 

<.0001 

1.33570 


In this case, you don’t really care about R-square or the adjusted R-square. The only table that you are 
going to focus on in this output is the Parameter Estimates table. In particular, you are interested only in 
VIF values, which give you an indication about the multicollinearity. None of the variables has a VIF value 
of more than 5, which indicates that there is no interdependency in the independent variables. So, you can 
keep all the independent variables while building the logistic regression model. 
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Predicting Delinquency 

You can see a note in the output saying "Probability modeled is SeriousDlqin2yrs='0,” which means the 
resultant probability from the model will be the probability of good. This would mean the higher the 
probability, the better the customer. If you want to predict the probability of bad instead of the probability 
of good, you have to use the descending option in the PROC LOGISTIC code. Most of the time, you try to 
predict the probability of the default rather than the probability of the nondefault. The following is the code 
for the final model. Remember, this will change the coefficients as well. 

proc logistic data=loans_data descending; 

model SeriousDlqin2yrs = utilization Age Num_loans Num_dependents Monthlylncome 
Num_Savings_Acccts DebtRatio; 

run; 


Table 11-25 shows the output of this code. 


Table 11 -25. Output of proc logistic on loans_data 

The SAS System 


The LOGISTIC Procedure 
Model Information 
Data Set 

Response Variable 
Number of Response Levels 
Model 

Optimization Technique 


WORK.LOANS_DATA 

SeriousDlqin2yrs 

2 

binary logit 
Fisher’s scoring 



Number of Observations Read 

150000 

Number of Observations Used 

150000 


Response Profile 

Ordered Value 

SeriousDlqin2yrs 

Total Frequency 

l 

l 

10026 

2 

0 

139974 


Probability modeled is SeriousDlqin2yrs=‘l’. 


Model Convergence Status 

Convergence criterion (GCONV=lE-8) satisfied. 
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Model Fit Statistics 

Criterion 

Intercept Only 

Intercept and Covariates 

AIC 

73618.167 

57641.254 

SC 

73628.085 

57720.601 

-2LogL 

73616.167 

57625.254 


Testing Global Null Hypothesis: BETA=0 



Test Chi-Square 

DF 

Pr> ChiSq 


Likelihood Ratio 

15990.9128 

7 

<.0001 

Score 

14078.3178 

7 

<.0001 

Wald 

11514.4825 

7 

<.0001 


Analysis of Maximum Likelihood Estimates 


Parameter 

DF 

Estimate 

Standard Error Wald Chi-Square 

Pr > ChiSq 

Intercept 

l 

-1.1431 

0.0622 

337.3241 

<.0001 

utilization 

l 

1.9713 

0.0312 

3999.3621 

<.0001 

Age 

l 

-0.0108 

0.000832 

167.2586 

<.0001 

Num loans 

l 

0.2464 

0.0106 

538.4191 

<.0001 

Num dependents 

l 

0.1941 

0.00960 

408.6747 

<.0001 

Monthlylncome 

l 

-0.00043 

5.983E-6 

5078.5897 

<.0001 

Num Savings Acccts 

l 

0.00945 

0.00517 

3.3408 

0.0676 

DebtRatio 

l 

1.1702 

0.0510 

526.4205 

<.0001 


Odds Ratio Estimates 

Effect 

Point Estimate 

95% Wald Confidence Limits 


utilization 

7.180 


6.755 

7.633 


Age 

0.989 


0.988 

0.991 


Num loans 

1.279 


1.253 

1.306 


Num dependents 

1.214 


1.192 

1.237 


Monthlylncome 

1.000 


1.000 

1.000 


Num Savings Acccts 

1.009 


0.999 

1.020 


DebtRatio 

3.223 


2.916 

3.561 
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Association of Predicted Probabilities and Observed Responses 


Percent Concordant 

82.8 

Somers’ D 

0.664 

Percent Discordant 

16.5 

Gamma 

0.668 

Percent Tied 

0.7 

Tau-a 

0.083 

Pairs 

1403379324 

c 

0.832 


Now you can observe that the note has changed in the SAS output. If a customer scores high in this 
model, then she has a higher probability to default in the repayment of the loan. The following is the note, 
reproduced from the SAS output in Table 11-25. 

Probability modeled is SeriousDlqin2yrs='l’. 

Goodness of Fit Statistic 

The concordance is 82.8 percent, and the discordance is 16.5 percent, which is really good for any logistic 
regression model. In real-life business problems, generally it is desirable to have a concordance of 75 percent 
or more. You can go ahead and use this model for prediction with greater confidence. 

Insignificant Variables 

Before you use this model for predictions, you have to see whether there are any independent variables 
that have no significant impact on the dependent variable. You can take a look at the P-value of individual 
independent variables (Table 11-26) to get an idea on their importance in the model. Any variable with a 
P-value of more than 5 percent in the Chi-square test is insignificant in the model. The predictions using this 
model will not be affected at all by keeping or removing such variables. 


Table 11 -26. Variable Coefficient Estimates 


Analysis of Maximum Likelihood Estimates 

Parameter 

DF 

Estimate 

Standard Error 

Wald Chi-Square 

Pr> ChiSq 

Intercept 

l 

-1.1431 

0.0622 

337.3241 

<.0001 

utilization 

l 

1.9713 

0.0312 

3999.3621 

<.0001 

Age 

l 

-0.0108 

0.000832 

167.2586 

<.0001 

Num loans 

l 

0.2464 

0.0106 

538.4191 

<.0001 

Num dependents 

l 

0.1941 

0.00960 

408.6747 

<.0001 

Monthlylncome 

l 

-0.00043 

5.983E-6 

5078.5897 

<.0001 

Num Savings Acccts 

l 

0.00945 

0.00517 

3.3408 

0.0676 

DebtRatio 

l 

1.1702 

0.0510 

526.4205 

<.0001 
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In this data set, the number of savings accounts has no significant impact on the delinquency of a 
customer (P-value = 0.0676). Table 11-27 shows the goodness of fit statistics keeping the number of savings 
account variable in the model. 

Table 11 -27. Goodness of Fit Statistics 

Percent Concordant 82.8 
Percent Discordant 16.5 


Let's see how the accuracy will change if you drop this variable from the final model. Ideally there 
should be no effect on the accuracy of the model because the variable is insignificant. 

/* Final Model after Dropping Num_Savings_Acccts */ 
proc logistic data=loans_data descending; 

model SeriousDlqin2yrs = utilization Age Num_loans Num_dependents Monthlylncome DebtRatio; 

run; 


The following is the output, which indicates there is absolutely no change in accuracy even after 
dropping the number of savings accounts variable. 

Percent Concordant 82.8 

Percent Discordant 16.5 


Final Model 

The following is the code for creating the final logistic regression model: 
proc logistic data=loans_data descending; 

model SeriousDlqin2yrs = utilization Age Num_loans Num_dependents Monthlylncome DebtRatio; 

run; 

Table 11 -28 is the output for the previous code. 


Table 11 -28. Output of proc logistic after Dropping Num_SavingsAcccts 


The SAS System 

The LOGISTIC Procedure 


Model Information 


Data Set 

WORK.LOANS DATA Response Variable SeriousDlqin2yrs 

Number of Response 

2 

Levels 


Model 

binary logit 

Optimization Technique 

Fisher's scoring 
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Number of Observations Read 

Number of Observations Used 

150000 

150000 


Response Profile 

Ordered Value 

SeriousDlqin2yrs 

Total Frequency 

l 

l 

10026 

2 

0 

139974 


Probability modeled is SeriousDlqin2yrs='l’. 


Model Convergence Status 

Convergence criterion (GCONV=lE-8) satisfied. 

Model Fit Statistics 

Criterion Intercept Only Intercept and Covarlates 

AIC 73618.167 57642.594 

SC 73628.085 57712.023 

-2 Log L 73616.167 57628.594 


Testing Global Null Hypothesis: BETA=0 
Test Chi-Square DF Pr > ChiSq 

Likelihood Ratio 15987.5724 6 <.0001 

Score 14075.3877 6 <.0001 

Wald 11511.9253 6 <.0001 


Analysis of Maximum Likelihood Estimates 

Parameter 

DF 

Estimate 

Standard Error 

Wald Chi-Square 

Pr > ChiSq 

Intercept 

1 

-1.0930 

0.0559 

382.7749 

<.0001 

utilization 

1 

1.9707 

0.0312 

3997.7947 

<.0001 

Age 

1 

-0.0108 

0.000831 

167.3011 

<.0001 

Num loans 

1 

0.2464 

0.0106 

538.4647 

<.0001 

Num dependents 

1 

0.1941 

0.00960 

408.8748 

<.0001 

Monthlylncome 

1 

-0.00043 

5.983E-6 

5078.5652 

<.0001 

DebtRatio 

1 

1.1706 

0.0510 

526.7798 

<.0001 
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Odds Ratio Estimates 


Effect 

Point Estimate 

95% Wald Confidence Limits 

utilization 

7.175 

6.750 

7.627 

Age 

0.989 

0.988 

0.991 

Num loans 

1.279 

1.253 

1.306 

Num dependents 

1.214 

1.192 

1.237 

Monthlylncome 

1.000 

1.000 

1.000 

DebtRatio 

3.224 

2.917 

3.563 


Association of Predicted Probabilities and Observed Responses 


Percent Concordant 

82.8 

Somers' D 

0.664 

Percent Discordant 

16.5 

Gamma 

0.668 

Percent Tied 

0.7 

Tau-a 

0.083 

Pairs 

1403379324 

c 

0.832 


The following is the final check list: 

• Applicability. A linear regression model can’t be applied for this data because the 
response variable is categorical (0 and 1). Logistic regression is a perfect fit for this 
dependent variable, which is binary in nature. 

• Chi-square test-. The P-value of the overall model Chi-square test is less than 
5 percent, so the model can be termed significant. 

• VIF-. None of the variables has a VIF of more than 5 percent, so there is absolutely no 
multicollinearity. 

• Concordance: The model concordance is 82.8 percent, which indicates that the 
resultant model is going to be a good predictor. 

• Individual variable impact: The P-values for all the independent variables 
(finally included in the model) are less than 5 percent, which indicates there 
are no insignificant variables in the final model. One insignificant variable was 
already dropped. 


Final Model Equation and Prediction Using the Model 

Here is the final model after substituting all the coefficients: 


P(y/x)= 


„ A> +A *1 +A*2 + A%+A*4 +'. +Pp *p 


1 + e 


o+a*, +aa +aa +a*« +..+/y 


^-1.0930L9707+(-0.0108Mge)+(0.2464‘Mtm Loans)+(0.1MlNum Dependent^- 0.00043* income)+(1.1706’Debt Ratio) 
P( Default ) | ^-1.09301.9707+(-0.0108*Age)+(0.2464*Num Incurs 11 (0.1941 Shim Dependent ) i( 0.00043* lucrum 1 } I (1.1 70(i'I)ehl Ratio) 
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Predictions Using the Model 

The bank can use the final model for quantifying the risk associated with each customer at the time of loan 
approval. The model gives the probability of default, so the bank can set a minimum limit, which may be that 
if any customer has a probability of default greater than 50 percent, as given by this model, the application 
will be rejected. 

For example, consider the three customers shown in Table 11-29 with the details furnished in their loan 
applications. 


Table 11 -29. Details in the Loan Application for Example Customers 



Tom 

David 

Hanks 

utilization 

30% 

70% 

70% 

Age 

34 

40 

50 

Num loans 

2 

5 

5 

Num dependents 

3 

4 

4 

Monthlylncome 

8000 

5000 

2500 

DebtRatio 

20% 

45% 

55% 


You can get the calculated probability values (Table 11-30) by substituting these values in the logistic 
regression equation: 


P [Default) 


e 


-1.09301.9707+(-0.0108Mge)+(0.2464*Num Loans )+(0.1941Mi/n Dependent)+(- 0.00043* Income )+(1.1706*Debt Ratio) 


1 + e 


-1.09301.9707+(—0.0108Mge)+(0.2464*iVum Loans )+(0.1941Num Dependent )+(■-0.00043* Income )+(1.1706*Debt Ratio) 


Table 11 -30. The Predicted Default Probabilities 


Customer 

Predicted Default Probability 

Decision 

Tom 

0.0047 (5%) 

Approve the loan 

David 

0.56 (56%) 

Reject the loan application 

Hanks 

0.79 (79%) 

Reject the loan application 


So, only Tom gets a loan. Generally, banks might not reject all the customers with a greater than 50 
percent probability. They might approve lesser value loans for the risky customers or charge higher interest 
rates and reduce the loan amount. Table 11-31 might be an example of a moderated implementation of the 
final defaulter model. 


Table 11-31. Decision Making Using the Default Probability Range 


Default Probability Range 

Decision 

0%-45% 

Approve loan application 

45%-65% 

Approve loan application with 


reduced loan amount 

65% 

Decline the loan application 
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Banks will send this model and the decision table to their front-end teams or install the model in a 
centralized system, which can be accessed by its loans processing team. It will be used as a primary tool for 
assessing any loan application. The decision methodology is scientific and reasonably accurate and uses 
analytics on the associated customer data. 


Conclusion 

In this chapter, we discussed that linear regression is not applicable for binary dependent variables. 

A logistic curve is best suited for such dependent variables. If the dependent variable has only two levels, you 
need to use logistic regression, which is also known as binomial logistic regression. If the response variable 
has more than two levels, such as good/bad/indeterminate, win/loss/draw, or satisfied/dissatisfied/neutral, 
then you need to use multinomial logistic regression. 

You learned how to fit a logistic regression line using SAS and also observed some of the similarities 
and differences in the output (with linear regression). The logistic regression line also suffers from 
multicollinearity issues; however, the process of treating multicollinearity is the same as in linear regression. 
There are other nonlinear regressions as well. Logistics is not the only nonlinear regression. You have 
finished the regression-related topics with this chapter and will get into other methods of modeling such as 
ARIMA in the following chapters. 
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CHAPTER 12 


Time-Series Analysis and 
Forecasting 



In this chapter, we introduce the concept of time-series forecasting straight using a couple of examples. 

We’ll talk about a popular bank, which has been offering credit cards to its customers for more than a 
decade. Every month the bank has some customers who file for bankruptcy. Some customers have defaulted 
on their loans for more than six months, and finally they need to be taken off the books. Obviously, this is a 
loss to the bank, in terms of dollars and accounts. Can the bank proactively estimate these losses for each 
of its portfolios? Can the bank proactively take necessary steps to control the loss and find out whether it is 
going to soar in the future? How does the bank forecast such losses based on historical data? 

Let's consider another example. A contact center or business process outsourcing (BPO) process 
receives and processes thousands of calls every month. The number of calls is not the same every day. 

The call volume is different depending upon whether it’s a weekend or a weekday and whether it’s morning 
or evening. Can the BPO center forecast the call volumes so that it can better manage the resources and 
employee shifts? Again, one question arises: how do you forecast the future call volumes? 

Another example is a laptop computer manufacturing company that has been in business for the last three 
decades. It has a couple of new models that will be launched in the next few months. Can the company determine 
the total addressable market? Can it estimate the sales of the next two quarters based on historical data? 

Many companies track variables related to the economy such as gross domestic product (GDP), growth 
rate index, unemployment rate, and so on. Based on historical values, can you forecast the future values of 
these variables? If yes, everyone would like to have those numbers. 

Similarly, can you forecast a stock price based on the past several years of data? Can you forecast the 
number of visitors for a web site? How do you forecast the number of viewers for a TV program? 

There are many such business scenarios that involve forecasting variables that vary with time. The 
forecasting time lines for the variables can be days, months, quarters, years, or even seconds and milliseconds. 
Basically, it can be any unit that represents a time line. You can use time-series analysis and forecasting 
techniques for most of these business scenarios. These techniques are the topic of discussion in this chapter. 


What Is a Time-Series Process? 

Any process that varies over time is a time-series process provided the interval is fixed. If you are talking 
about monthly data and one month is the interval between two time points, then you should have one value 
of the variable every month for it to be defined as a time-series process. 

Time-series data is a sequence of records collected from a process with equally spaced intervals in 
time. In simple terms, any metric that is measured over time is a time-series process, given that the time 
interval is the same between any two consecutive points. In the following text, you will consider some 
time-series plots as examples. 
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You denote a time series with Y t , where Y is the value of the metric at time point t. Refer to Table 12-1, 
which is charted in Figure 12-1. It is an example of the daily call volume of the last two years for a 
voice-processing BPO. 


Table 12-1. Daily Call Volume of a Voice-Processing BPO 


Day 

Denoted by 

Call Volume 

10/25/2014 

Y t 

99,552 

10/24/2014 

Y t , 

113,014 

10/23/2014 

Y , 2 

118,502 

10/22/2014 

Y,.3 

119,437 

10/21/2014 


123,410 

10/20/2014 


129,368 

10/19/2014 


135,961 

10/18/2014 


131,698 

2/23/2014 

Y 

t-p 

113,214 


Call Volumes Data 

160000 



40000 

20000 

0 

2 / 23/2014 3 / 23/2014 4 / 23/2014 5 / 23/2014 6 / 23/2014 7 / 23/2014 8 / 23/2014 9 / 23/2014 10 / 23 / 20 : 

Date 

Figure 12-1. Daily call volume for a voice-processing BPO 
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Figure 12-2 shows an example of time-series data for monthly laptop computer sales. 


Laptop Sales 
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Figure 12-2. Time-series data on monthly laptop sales 

Figure 12-3 shows the time-series chart of the United States’ GDP for the past 50 years. This GDP is 
calculated at purchasers’ prices. It is the sum of the gross value added by all resident producers in the 
economy. Product taxes and subsidies are treated appropriately while calculating the value of the products. 
It’s not necessary to get into details of how GDP is actually calculated. 
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USA GDP 


18,000 



Year 


Figure 12-3. Time-series data on U.S. GDP (Source: World Bank, http://data.mrldbank.org/ 
country ?display=graph ) 

Figure 12-4 shows the time-series chart of U.S. mobile cellular subscriptions (per 100 people). 

Mobile cellular subscriptions (per 100 people) 
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Figure 12-4. U.S. mobile cellular subscriptions, per 100 people (Source: World Bank, http: //data. 
mrldbank.org/indicator/IT. CEL. SETS.P2 ) 
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Main Phases of Time-Series Analysis 

So far in this chapter, you have seen several examples of time-series data. The aim of time-series analysis is 
to comprehend the historical time line of data, analyze it to uncover hidden patterns, and finally model the 
patterns to use for forecasting. There are three main phases of time-series analysis (Figure 12-5). 

1. Descriptive analysis 

2. Modeling 

3. Forecasting the future values 



Forecasting 


Figure 12-5. Main phases in time-series analysis 


In the descriptive phase, you try to understand the nature of the time series. You try to answer questions 
about what the trend looks like, upward or downward, and whether there any seasonal trend in the process. 
Finding patterns in the data is a vital step. In the modeling or pattern discovery step, you model the inherent 
properties of the time-series data. There are several methods of finding patterns in the process of time-series 
analysis. 

Once you have identified the patterns, forecasting is relatively easy. It is simple to use a model to 
forecast future values. In the next section, you will see some modeling techniques. Before applying these 
techniques, it helps if you get a basic feel of the data using descriptive statistics methodologies. You have 
already studied predictive modeling in earlier chapters. Normally, in predictive modeling, you predict the 
dependent variable Y using a set of X variable values. In time-series analysis, it’s different. Here you predict 
Y using the previous values of Y. The following mathematical representation helps further clarify this 
difference: 


Predictive modeling Y versus X t , X 2 ,... X p 

Time-series analysis Y versus Y (Y t versus Y tl ) 


Modeling Methodologies 

There are several techniques for modeling a time series, both simple and heuristic. Mainly all these 
modeling techniques can be classified into two categories: heuristic and proper statistical methods. There 
are some major drawbacks in simple heuristic or instinctive methods. There are hardly any statistical tests 
to back up simplistic heuristic methods. That is why, in this book, the discussion is limited to more scientific 
statistical methods. The main focus will be on the Box-Jenkins approach (or ARIMA technique), which is 
probably one of the most commonly used techniques in the time-series domain. 


445 










CHAPTER 12 TIME-SERIES ANALYSIS AND FORECASTING 


Box-Jenkins Approach 

George Box and Gwilym Jenkins popularized the auto-regressive integrated moving average (ARIMA) 
technique of modeling a time series. ARIMA methods were not actually invented by Box and Jenkins, but 
they made some significant contributions in simplifying the techniques. To understand model building 
using the Box-Jenkins approach, you first need to develop an understanding of terms like ARIMA, AR, MA, 
and ARMA process. 


What Is ARIMA? 

Time-series models are known as ARIMA models if they follow certain rules. One such rule is that the values 
of the time series should fall under certain limits. Later sections of the chapter will come back to ARIMA. 


The AR Process 

An auto-regressive (AR) process is where the current values of the series depend on their previous values. 
Auto-regression is also an indicative name, which specifies that it is regression on itself. The auto-regressive 
process is denoted by AR(p), where p is the order of the auto regressive process. In the AR process, the 
current values of the series are a factor of previous values. P determines on how many previous values the 
current value of the series depends. 

For example, consider Y t , Y t ] , Y t 2 , Y , Y t 4 , Y s , Y t . 6 ,... Y t k , which constitutes a time series. If this series is 
an AR(1) series, then the value ofY at the given time t will be a factor of its value at time t-1. 

Y t = <Y t . 1 + g t 

is the factor or the quantified impact Y t | on Y t , and s t is the random error at time t. s t is also 
known as white noise. When I say that Y at any given time t is a factor of its value at time t-1, it may not 
exactly be equal to a *Y . It may be slightly higher or lower than al*Yt-l. But you can’t precisely tell what 
will be the value of s ( . It is just a small random value at time t, which may be positive or negative. This 
term (s t ) can’t be specifically quantified in an actual time-series model because it takes different values at 
different time points. 

For an AR(2) series, the value of Y at time t will be a factor of the value at time t-1 and t-2. 


Y, “ a ,*Y t ] + a 2 *Y ( 2 + e t 

a 1 is the factor or the quantified impact Y 3 on Y , and a 2 is the quantified impact Y on Y . 
Likewise, for a series to be AR(P), Y will depend on the previous Y values until the time point t-p. 


a 2 , a 2 , a 3 ,_a p are the quantified impacts of Y l ,Y t 2 ,... Y t on Y ( 

Let’s look at an example in the sales domain. If you come across a sales series where the current month's 
sales depends upon the previous month’s sales, then you call that an AR(1) series. 

The following equation and the plot in Figure 12-6 represent an AR(1) process for time-series sales data: 


Y t - 0.76*Y tl + s, 
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Sales 
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Figure 12-6. A typical AR (1 ) time-series equation plot (product sales over time) 

The following equation and the plot in Figure 12-7 represent an AR(2) process of time-series sales data 
for a different product: 

Y t = 0.45*Y tl + 0.48*Y 2 + s t 
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Figure 12-7. A typical AR (2) time-series equation plot (product sales over time) 
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Please note that Figures 12-6 and 12-7 are typical examples, but not every AR(1) or AR(2) graph will look 
like this. The shape of a time-series plot varies from one equation to another even within the same order of 
an AR(p) process. It is not easy to determine the order of an AR process by simply looking at the shape of the 
time-series curve alone. 


The MA Process 

A moving average (MA) process is a time-series process where the current value and the previous values 
in the series are almost the same. But the current deviation in the series depends upon the previous white 
noise or error or shock (that is, s t , K t | , and so on). Generally, E t is considered to be an unobserved term. It 
can be a small random change in the current series because of some unknown reason; it maybe due to some 
external effect. For example, while studying a sales trend, a small observed random change in sales might 
be due to external factors such as accidental rain, snow, or exceptionally hot weather. In an AR process, Y 
depends on the previous values of Y, whereas in the MA process, Y t ] doesn’t come into picture at all. It’s all 
about the short-term shocks in the series; MA is not related to the long-term trends, like in an AR process. In 
the MA process, the current value of the series is a factor of the previous errors. MA(q) is the analog of AR(p) 
as far as the notation for the MA process is concerned. Here q determines how many previous error values 
have an effect on the current value of an MA series. s t , e t , e ,..., £ t k denotes the errors or shocks or the 
white noises at time t, t-1,_t-k. 

If a series is MA(1), then the deviation at time t will be a factor of the error at t and t-1. 

Y t -p = E, + b i*E,i 


In the previous equation: 

• bj is the factor or the quantified impact of s t 1 on the current deviation. 

• p is the mean of the overall series. 

• Y t - p is the deviation at time t. 

Note that there is no Y t ] term in this equation. The current error and previous error constitute only the 
right side of the equation. 

If a series is MA(2), then the deviation at time t will be a factor of the errors at t, t-1, and t-2. 

Y t -P = £ t + b iVVs t -2 


In the previous equation: 

• bj and b 2 are the factors or the quantified impacts of st-1 and st-2 on the 
current deviation. 

• p is the mean of the overall series. 

• Y t -pis the deviation at timet. 

Similarly, for an MA(q) series, the deviation at time t will be a factor of the errors at t, t-1,.t-p. 

Y t “P = E t + b i*E t 1+ b 2*S t _ 2 .+ b q *E t q 

You will appreciate that an AR component is a long-term trend component, while an MA component is 
a short-term component. 

The following equation and the plot in Figure 12-8 represent an MA(1) process for time-series sales data: 

Y t = E t +0.3\ 1 
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Figure 12-8. A typical MA(1) time-series equation plot (product sales over time) 

The following equation and the plot in Figure 12-9 represent an MA(2) process for time-series sales data: 

Y t = 8 t + 0.1*e t i+0.3*£ t2 
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Figure 12-9. A typical MA(2) time-series equation plot (product sales over time) 
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As discussed for an AP(p) series, the shape of any MA(q) series curve also depends upon the actual 
parameters. Even if the order of two MA(q) series is the same, their plot shapes can differ based upon the 
actual parameter values in their respective equations. 


ARM A Process 

If a process shows the properties of an auto-regressive process and a moving average process, then it is 
called an ARMA process. In an ARMA time-series process, the current value of the series depends on its 
previous values. The small deviations from the mean value in an ARMA process are a factor of the previous 
errors. You can think of an ARMA process as a series with both long-term trend and short-term seasonality. 
ARMA(p,q) is the general notation for an ARMA process, where p is the order of the AR process and q is the 
order of the MA process. 

For example, if Y , Y , Y , Y , Y , Y , Y ,., Y k are the values in the time series and 

e„ e , s t2 ,..., s tk are the errors at time t, t-1, t-2,_t-k, respectively, then an ARMA(1,1) series is written as 

follows: 


Y t = a i*Y,_i + s t + b iXi 

a 1 is the factor or the quantified impact of Y ( on Y , and s t is the random error at time t. bj is the factor or the 
quantified impact of e w on the current value. 

An ARMA(2,1) series is written as follows: 


Y = a,*Y , + a *Y „ + s. + b,*s. 


a 1 and a 2 are the quantified impacts of Y ; and Y on Y, and s t is the random error at time t. ly is the factor or 
the quantified impact of s t l on the current value. 

In the same way, the generic ARMA (p,q) series is written as follows: 


Y, = a,*Y,, + a *Y,, + a,*Y,, + . 


..+ a *Y. + s, + b,' 

p t-p t i 


,+ b *e,„ .+ b *e, 

•1 2 t-2 q t-q 


a,,a„, a,,_a are the quantified impacts of Y , ,Y _Y on Y, and b., h,_b are the factors or the 

1 2 3 p 1 r t-1 t-2 t-p t 1 2 q 

quantified impacts of e ( ] and s t 2 ... s on Y t . 

The equation and the plot in Figure 12-10 represent an ARMA(1,1) process of a typical sales time-series 
data: 


Y t = 0.75* Y tl + e, + 0.2* £ (1 
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Figure 12-10. A typical ARMA( 1,1) time-series equation plot (product sales over time) 

The following equation and the plot in Figure 12-11 represent an ARMA(2,1) process for time-series 
sales data: 

Y t = 0.35* Y (] + 0.45*Y Y t 2 + s t +0.1* e tl 
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Figure 12-11. A typical ARMA(2,1) time-series equation plot (product sales over time) 
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Now you know what an ARMA process is. And you are also in a position to envisage the equation of a general 
ARMA(p,q) process. You are now ready to learn the Box-Jenkins ARIMA approach of time-series modeling. 

Understanding ARIMA Using an Eyesight Measurement 
Analogy 

I’ll use an analogy related to eyesight measurement and prescription eyeglasses as an example to explain a 
serious concept. Nowadays there is a lot complicated equipment to measure human eyesight. It is relatively 
easy to accurately measure eyesight and get the right prescription for eyeglasses. But a few decades back, 
before today’s sophisticated and computerized eye testing machines, doctors accomplished this task 
manually. Today, almost everyone who has visited an eye doctor may easily recognize the picture presented 
in the Figure 12-12. It is known as a Snellen chart (though this name is not as common as the chart itself). 
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Figure 12-12. A Snellen chart (courtesy: Wikipedia) 

To test eye sight and prescribe eyeglasses, doctors perform a small test. Instead of using special 
equipment, in the past doctors had a box full of lenses (of different powers). The patient was asked to sit in 
a chair and was given an empty frame to put on her eyes. The doctor used to put differently powered lenses, 
on by one, in the frame and asked the patient to read from the Snellen chart. Some patients, for example, 
read the top seven rows and struggled with the lower ones. The doctor then removed the first lens and put 
another. After much such iteration, the doctor used to finalize on the exact lenses to be used in the patient’s 
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glass. Some patients got diagnosed as nearsighted and some with farsightedness. The basic assumption in 
this process was that all the patients are literate. But what if the patient was illiterate and couldn’t read any 
letters? 

So, the major steps in the process of eyesight determination and prescription can be listed as follows: 

1. Assume that patient is literate. 

2. Based on some tests, identify nearsightedness or farsightedness and get a rough 
estimate of eyesight. 

3. Estimate the exact eyesight by trying various lenses. 

4. Use the test results to give the prescription. 

A similar analogy can be used with Box-Jenkins approach. Say you have a time series in your hands 
and you want to forecast some future values in this series. You first need to identify whether the time-series 
process is an AR process or an MA process or an ARMA process. You also need to identify the orders, p and q, 
of AR(p) or MA(q) or ARMA(p,q) processes as applicable. Once you identify the type of series and the orders, 
you can attempt to write the series equation. You are already familiar with the AR(p), MA(q), and ARMA(p,q) 
equations. The next step is to find parameters such as a v a 2 ,... a p , and b 1( b 2 ,... b (| as applicable. Before 
you move on to identifying the model, there is an assumption to be made; the series has to be stationary 
(in the coming sections, I explain more about stationary time series). This is to simplify the overall model 
identification process. Table 12-2 uses the eyeglasses prescription analogy to illustrate the Box-Jenkins 
approach. After that I discuss various steps involved in the Box-Jenkins approach. 


Table 12-2. An Analogy Between a Vision Test and the Box-Jenkins Approach 


Vision Test 

Box-Jenkins Approach 

Assume the patient is literate. 

Assume that the time series is stationary; otherwise, 
make it stationary. 

Based on some tests, identify nearsightedness 
or farsightedness and get a rough estimate of 
eyesight. 

Based on plots (ACF and PACF functions, explained later 
in this chapter), identify whether the model is an AR or 

MA or ARMA process. 

Estimate the exact eyesight by trying various 
lenses. 

Estimate the parameters such as a , a 2 ,... a , and 
b , b 2 ,... b . 

1 2 q 

Use the test results to give the prescription. 

Use the final model for forecasting. 


Steps in the Box-Jenkins Approach 

Once again I’ll show a time-series forecasting problem. Consider some time-series data of a premium stock 
over a period of time. Let’s assume you have the stock price data for the past year. Also assume that you 
want to predict the stock prices for the next week. You will use the Box-Jenkins approach for this forecasting. 
First you need to make sure that the stock price time-series process is stationary. Then you need to identify 
the type of process, which approximates the pattern followed by the stock price data. Is it an AR process or 
an MA process or an ARMA process? Once you have the basic model equation in place, you can estimate 
the parameters. Successfully completing all these steps concludes the model-building process. You now 
have the final equation that can be used for forecasting the future values of the stock under consideration. 
You also need to take a look at the model accuracy or the error rate before you can continue with the final 
forecasting and model deployment. What follows is a detailed explanation of each step. 
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Step 1: Testing Whether the Time Series Is Stationary 

If a time-series process is stationary, it’s much easier to build the model using the Box-fenkins methodology. 

What Is a Stationary Time Series? 

"A time series is said to be stationary if there is no systematic change in mean (no trend), if there is no 
systematic change in variance, and if strictly periodic variations have been removed," according to Dr. Chris 
Chatfield in The Analysis of Time Series: An Introduction (Chapman and Hall, 2003). A stationary time series 
is in a state of statistical equilibrium. If the series is varying within certain limits and if the previous or future 
values of the mean and variance of the series are not going to be exceptionally high, then the series can be 
understood as a stationary series. In the stationary time-series process, the mean and variance hover around 
a single value. With the growth of a series, if the mean and variance of the series also tend to grow extremely 
high, then the series is considered to be nonstationary. It’s difficult to identify patterns in a nonstationary 
time series, and model building can be tough. A nonconstant mean or nonconstant variance is a sign of a 
nonstationary time-series process (Figure 12-13). These processes are also called explosive. 

Non-Stationary Time Series 


35000 


_= 

ro 

> 



Figure 12-13. An example of a nonstationary time series 

In some cases, it is easy to identify a nonstationary time series. You need to ask two questions: 

• What will the series mean be at time oo (infinity)? 

• What will the series variance be at time oo (infinity)? 
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If the answer to either one of these questions is <*>, then you can safely conclude that the series under 
consideration is nonstationary. As a nonstationary series grows, you can see that the values of the series 
start inflating too much; in fact, they tend to go out of proportion soon. It becomes really tough to predict 
the series values using the Box-Ienkins approach in such a case. Real-life scenarios may encounter many 
nonstationary series. You may have to transform or differentiate to make them stationary. Later sections will 
cover the details of series transformation and differentiation. 

Figure 12-14 shows an example plot of a stationary time series. 

Stationary Time Series 



Figure 12-14. An example of a stationary time series 

In Figure 12-14, some steadiness is apparent in the series plot. It's not inflating too much from its mean 
value line. In this case, the value of variance or mean will also show steadiness or a state of equilibrium. 

By now I have discussed many times that simply by looking at the curve shape, it is not a good idea to come 
to a conclusion. You need a scientifically proven statistical approach to test whether a series is stationary. 

In fact, curve shapes can be misleading at times and can lead to wrong conclusions. For example, let’s 
take the series shown in Figure 12-15, which represents weekly fluctuations in Microsoft’s stock price. 

The data shown in this plot is Microsoft’s stock prices from January 2010 to September 2014 (Source: 
http: //finance. yahoo. com/echarts ?s=MSFT). Just looking at the curve shape, you can’t undoubtedly tell 
whether it is a stationary or nonstationary series. You need a solid test for testing stationarity. 
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Microsoft Stock Price 



Time Points 


Figure 12-15. Time-series data for weekly stock prices of a listed company 


Testing Stationarity Using a DF Test 

To test the stationarity of a series, you use Dickey Fuller (DF) test checks. The actual theory behind DF and 
Augmented Dickey Fuller (ADF) tests is complicated and beyond the scope of this book. What follows is a 
simplified and practical use of a DF test, which may be sufficient for field use. 

The null and alternative hypotheses of a DF test are as follows: 

• H q : The series is not stationary. 

• H,: The series is stationary. 

So, you perform a DF test and take note of the P-value. Looking at the P-value of the DF test, if the 
P-value is less than 5 percent (0.05), you reject the null hypothesis, which is equivalent to rejecting the 
hypothesis that the series is not stationary. So, a series is concluded as stationary when the P-value of a DF 
test is less than 0.05. On the other hand, if the P-value is more than 0.05, then you go ahead and accept the 
null hypothesis, which means that the series is not stationary. 

SAS Code for Testing Stationarity 

The following is the SAS code for testing the stationarity of a series. Let’s assume that the Microsoft stock 
price data is imported into the SAS environment and stored in a data set called ms. 

proc arima data= ms; 

identify var= stock_price stationarity=(DICKEY); 

run; 


The following is an explanation of this code: 

• proc arima: arima is the procedure to fit ARIMA models. 

• data= ms;: The data set name is ms. 

• identify var= stock price: This mentions the time-series variable, which is 
Ad j_close in this case. 

• stationarity= (DICKEY): This is the keyword for the DF test. 
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Table 12-3 shows the output of this code. 

Table 12-3. Output ofPROC ARIMA on the Data Set ms 

(DF Test) 

The ARIMA Procedure 

Name of Variable = stock price 

Mean of Working Series 

29.20438 

Standard Deviation 

6.539502 

Number of Observations 

251 


Autocorrelations 

Lag 

Covariance 

Correlation 


198765432101234567891 

Std Error 

0 

42.765085 

1.00000 

1 

| ******************** | 

0 

l 

41.916461 

0.98016 

1 

|********************| 

0.063119 

2 

41.181417 

0.96297 

1 

|******************* | 

0.107885 

3 

40.429605 

0.94539 

1 

|******************* | 

0.137942 

4 

39.522281 

0.92417 

1 


0.161708 

5 

38.561047 

0.90169 

1 

|****************** | 

0.181535 

6 

37.544646 

0.87793 

1 

|****************** | 

0.198579 

7 

36.563693 

0.85499 

1 

|*«*************** | 

0.213483 

8 

35.643849 

0.83348 

1 

|***************** | 

0.226715 

9 

34.680732 

0.81096 

1 

|**************** | 

0.238611 

10 

33.800014 

0.79036 

1 

|**************** j 

0.249350 

11 

33.004501 

0.77176 

1 

|*************** | 

0.259139 

12 

32.306690 

0.75545 

1 

|*************** | 

0.268140 

13 

31.663022 

0.74039 

1 

j*************** | 

0.276489 

14 

30.943593 

0.72357 

1 

|************** | 

0.284279 

15 

30.128160 

0.70450 

1 

|************** | 

0.291524 

16 

29.475695 

0.68925 

1 

|************** | 

0.298230 

17 

28.873215 

0.67516 

1 

j************** | 

0.304510 

18 

28.196376 

0.65933 

1 

|************* | 

0.310417 

19 

27.582788 

0.64498 

1 

|************* | 

0.315947 

20 

27.015599 

0.63172 

1 

|************* | 

0.321150 

21 

26.431044 

0.61805 

1 

|************ 

0.326063 

22 

25.838137 

0.60419 

1 

j************ 

0.330697 

23 

25.265838 

0.59081 

1 

j************ 

0.335066 

24 

24.735666 

0.57841 

1 

|************ 

0.339191 


marks two standard errors 


457 











CHAPTER 12 TIME-SERIES ANALYSIS AND FORECASTING 


Inverse Autocorrelations 

Lag 

Correlation 

■1 987654321 01 234567891 

l 

-0.43201 

*********| | 

2 

-0.03023 

•1 ■ 1 

3 

-0.06586 

•1 ■ 1 

4 

0.01759 

• 1 • 1 

5 

-0.02020 

■ 1 • 1 

6 

0.02871 

■ r. i 

7 

0.02352 

■ i • i 

8 

-0.06275 

■ *i ■ i 

9 

0.03719 

■ r- i 

10 

0.00328 

■ i • i 

11 

0.03405 

■ r- i 

12 

0.01159 

■ i • i 

13 

-0.04483 

■ i ■ i 

14 

-0.07979 

■i ■ i 

15 

0.09398 

■ i". i 

16 

0.01896 

■ i • i 

17 

-0.05763 

■ *i ■ i 

18 

0.03131 

■ r. i 

19 

0.01035 

■ i • i 

20 

-0.02261 

■ i • i 

21 

-0.00559 

■ i • i 

22 

0.01411 

• i • i 

23 

-0.00034 

■ i • i 

24 

-0.00268 

■ i • i 
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Partial Autocorrelations 

Lag 

Correlation 

■1 987654321 01 234567891 

l 

0.98016 

|««****«*«»»*«*******| 

2 

0.05757 

■ I*- 1 

3 

-0.01432 

■ 1 • 1 

4 

-0.10292 

•1 ■ 1 

5 

-0.05609 

•1 ■ 1 

6 

-0.04894 

•1 ■ 1 

7 

0.01308 

■ 1 • 1 

8 

0.03603 

■ I*- 1 

9 

-0.02412 

■ 1 • 1 

10 

0.03221 

■ r- i 

11 

0.03873 

■ i*. i 

12 

0.05418 

■ i*. i 

13 

0.02240 

■ i • i 

14 

-0.06212 

■ *i ■ i 

15 

-0.09624 

•i ■ i 

16 

0.06381 

■ r- i 

17 

0.04601 

■ i*. i 

18 

-0.02960 

• *i ■ i 

19 

0.02061 

■ i • i 

20 

0.01711 

■ i • i 

21 

-0.01757 

■ i • i 

22 

-0.01194 

■ i • i 

23 

0.00998 

■ i • i 

24 

0.00511 

■ i • i 


Autocorrelation Check for White Noise 


To Lag 

Chi-Square 

DF 

Pr>ChiSq 

Autocorrelations 





6 

1338.63 

6 

<.0001 

0.980 

0.963 

0.945 

0.924 

0.902 

0.878 

12 

2356.83 

12 

<.0001 

0.855 

0.833 

0.811 

0.790 

0.772 

0.755 

18 

3147.49 

18 

<.0001 

0.740 

0.724 

0.705 

0.689 

0.675 

0.659 

24 

3768.57 

24 

<.0001 

0.645 

0.632 

0.618 

0.604 

0.591 

0.578 
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Augmented Dickey-Fuller Unit Root Tests 

Type 

Lags 

Rho 

Pr< Rho 

Tau 

Pr<Tau 

F 

Pr> F 

Zero Mean 

0 

0.5872 

0.8278 

1.27 

0.9484 




l 

0.5975 

0.8302 

1.41 

0.9610 




2 

0.6488 

0.8424 

1.61 

0.9740 



Single Mean 

0 

0.4367 

0.9749 

0.20 

0.9725 

0.81 

0.8654 


1 

0.8208 

0. 9852 

0.42 

0.9835 

1.00 

0.8151 


2 

0.9180 

0.9872 

0.49 

0.9862 

1.30 

0.7389 

Trend 

0 

-9.5408 

0.4616 

-2.35 

0.4071 

4.13 

0.3520 


1 

-8.4784 

0.5410 

-2.25 

0.4603 

4.23 

0.3321 


2 

-7.0116 

0.6587 

-1.93 

0.6390 

3.34 

0.5093 


The output in Table 12-3 includes several tables, but the one that matters here is the last one named 
Augmented Dickey-Fuller Unit Root Tests. Within that table you need to focus on the section named Single 
Mean. The Zero Mean portion of the table is calculated by assuming the mean of the series is zero. The 
Trend portion of the table is calculated by assuming the null hypothesis of a DF test is true, meaning that the 
series is stationary with some trend component. You can safely ignore these two portions of the output from 
your inference. So, to keep it simple, let's concentrate only on Table 12-4, which is taken from the complete 
table, Augmented Dickey-Fuller Unit Root Tests. 


Table 12-4. The Single Mean Portion of Augmented Dickey-Fuller Unit Root Tests 


Augmented Dickey-Fuller Unit Root Tests 

Type 

Lags 

Rho 

Pr < Rho 

Tau 

Pr<Tau 

F 

Pr> F 

Single Mean 

0 

0.4367 

0.9749 

0.20 

0.9725 

0.81 

0.8654 


l 

0.8208 

0.9852 

0.42 

0.9835 

1.00 

0.8151 


2 

0.9180 

0.9872 

0.49 

0.9862 

1.30 

0.7389 


You need to look at the P-value of a single mean for an augmented Dickey Fuller test. 

• Pr<Rho values are more than 5 percent for lags 0,1, and 2. 

• Similarly, PrcTau values are more than 5 percent for lagsO, 1, and 2. 

So, you safely conclude here that the series under consideration is not stationary. 


Note As discussed earlier, the exact explanation of a DF test is complicated and is beyond the scope 
of this book. The following are some notes for statistics enthusiasts. 
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The DF test has two sets of P-values: 

• The first set of P-values (Pr<Rho) is for testing the unit roots directly. Looking at these 
P-values (Table 12-4), you can conclude that the series is not stationary. 

• The second set of P-values (PrcTau) is for testing the regression of a differentiated Y 
on a constant, time t, Y , and differentiated Y . 

Achieving Stationarity 

If a series is not stationary like the way you saw in the previous case (Figure 12-15), you can differentiate it to 
make it stationary. If Y t is the original series, then AY t =Y t - Y t r Instead of taking the original series values, 
you consider the differentiated series, as shown in Table 12-5. 

Table 12-5. Differentiated Series Data 


Time 

Y t 

(Original Series) 

Y„ 

(Lag of Original Series) 

AY t = Y t -Y M 
(Differentiated Series) 

Jan 

200 

- 

- 

Feb 

250 

200 

50 

Mar 

230 

250 

-20 

Apr 

245 

230 

15 

May 

280 

245 

35 

Jun 

256 

280 

-24 


In the previous Microsoft example, the stock price time series was not stationary. Let’s create a 
differentiated time series using the following SAS code. It is pretty straightforward. 

data msl; 
set ms; 

diff_stock_price=stock_price-lagl(stock_price); 

run; 


msl is the new data set, Lagl denotes the Y ] values, and diff_stock_price is the new differentiated series. 
Table 12-6 is a snapshot of the differentiated series. 
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Table 12-6. A Snapshot of the Differentiated Stock Price Time Series 


Obs 

stock price 

diff stock price 

l 

27.05 


2 

27.23 

0.18 

3 

25.55 

-1.68 

4 

24.86 

-0.69 

5 

24.72 

-0.14 

6 

24.64 

-0.08 

7 

25.5 

0.86 

8 

25.41 

-0.09 

9 

25.34 

-0.07 

10 

25.94 

0.60 


Note that some series may not be stationary even after the first differentiation. You then need to go for 
the second differentiation. If even the second differentiation doesn’t work, you may have to try some other 
transformation logarithm. 

Now instead of considering the original series, in other words, the stock price (Y t ), you now consider the 
differentiated stock price and test it for stationarity. The following is the code to perform this test: 

proc arima data= msl; 

identify var=diff_stock_price stationarity=(DICKEY); 

run; 

Table 12-7 shows the output of this code. Again, you concentrate only on the Augmented Dickey-Fuller 


Unit Root Tests table of the output. The focus will be on the Single Mean portion. 

Table 12-7. Output of PROC ARIMA on msl (DFTest) 

Augmented Dickey-Fuller Unit Root Tests 

Type 

Lags 

Rho 

Pr< Rho 

Tau 

Pr<Tau 

F 

Pr> F 

Zero Mean 

0 

-270.879 

0.0001 

-17.17 

<.0001 




1 

-289.824 

0.0001 

-12.06 

<.0001 




2 

-265.730 

0.0001 

-9.31 

<.0001 



Single Mean 

0 

-272.508 

0.0001 

-17.26 

<.0001 

148.89 

0.0010 


1 

-296.605 

0.0001 

-12.19 

<.0001 

74.30 

0.0010 


2 

-280.934 

0.0001 

-9.46 

<.0001 

44.80 

0.0010 

Trend 

0 

-275.376 

0.0001 

-17.43 

<.0001 

151.98 

0.0010 


1 

-306.994 

0.0001 

-12.36 

<.0001 

76.36 

0.0010 


2 

-303.187 

0.0001 

-9.62 

<.0001 

46.34 

0.0010 


Both the sets of P-values on Rho and Tau are well below 5 percent. Hence, you conclude that the series 
diff_stock_price is stationary. 
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The Integration Component in the ARIMA Process 

In the previous section, you saw how to make a nonstationary process stationary by differentiating it. The 
original Microsoft price was in tens of dollars, whereas the differentiated value (the new series, which is 
stationary) is in fractions. Please refer to Table 12-6. 

The differentiated series is stationary, and it can be used for analysis using the Box-fenkins approach. 
The new series can be any ARMA(p,q) process. But the forecasted values of this series will be in fractions 
only. As a matter of fact, you have no interest in forecasting the differentiated time series. Our main interest 
is in forecasting actual Microsoft stock prices but obviously not the differentiated series. You can still derive 
the original series values from the differentiated series. You need to perform an integration act on the 
differentiated series, which is the mathematical opposite of differentiation. You made the series stationary 
by differentiating it. Now you apply the integration to the transformed series to bring it back into its original 
form. Here the order of the integration (d) is 1. The order of differentiation, which you did to the stock 
prices series, was also 1. An ARMA series with integration is also known as an ARIMA series. It is denoted by 
ARIMA (p,d,q), where p is the order of the AR component, d is the order of integration, and finally q is the 
order of the MA component. 

The order of integration required in the current example process is 1. This is simply because to make it 
stationary, you did the differentiation once. Some series are stationary without any need of differentiation. 
For such time series, the order of integration is 0. The following are some examples of the ARIMA processes: 
The ARIMA(1,1, 0) series is written as follows: 

AY t = a 1 *AY tl + g t 

a 1 is the factor or the quantified impact AY t ] on AY t , and s t is the random error at time t, in other words, 

Y-Y;- 

The ARIMA(2,1, 0) series is written as follows: 

AY t = a 1 *AY tl + a2*AY t2 + s t 

a 1 and a, are the factors or the quantified impacts of AY tl and AY t 2 on AY t , and g, is the random error at time 
t, in other words, AY=Y-Y . 

The ARIMA(1,1,1) series is written as follows: 

AY t = a 1 *AY (2 + s ( + b 1 *e (2 

a 1 is the factor or the quantified impact of AY tl on AY t , and g ( and g ( ] are the random errors at time t and t-1, 
in other words, AY = Y -Y . 

Integration After Forecasting 

You must be wondering how to integrate after forecasting. Table 12-6 shows the form of the series after 
differentiating the original series in the Microsoft stock price example. In this example, you found that the 
stock price was not stationary, but diff stock price is stationary. 

Now let’s assume you have gone ahead with the diffstockprice series and produced three 
forecasted values after the analysis. Refer to observations 11,12, and 13 in Table 12-8. 
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Table 12-8. Forecasted Values in the Time Series dijf stock price 


Obs 

stock price 

diff stock price 

l 

27.05 


2 

27.23 

0.18 

3 

25.55 

-1.68 

4 

24.86 

-0.69 

5 

24.72 

-0.14 

6 

24.64 

-0.08 

7 

25.5 

0.86 

8 

25.41 

-0.09 

9 

25.34 

-0.07 

10 

25.94 

0.60 

11 


-0.10 

12 


0.21 

13 


-0.11 


Having done this, how do you know what the stock prices will be for observations 11,12, and 13? 
In other words, the question here is, how do you bring back the forecasted series in its original units? 

You know that the differentiated stock prices at any given time t can be calculated by the following 
formula: 

Diff_stock_price at time t = (Stock price at time t) - (Stock price at time t-1) 

Before forecasting: 

Diff_stock_price at time 10 = (Stockprice at time 10) - (Stockprice at time 9) 
Diff_stock_price at time 10 = 25.94 - 25.34 
Diff_stock_price at time 10 = 0.60 

After forecasting: 

Diffstockprice at time 11 = (Stockprice at time 11) - ( Stockprice at time 10) 

Inserting the values in this equation from Table 12-8, you get the following: 

-0.10 = Stockprice at time 11-25.94 
Stockprice at time 11 = 25.94 -0.10 
Stockprice at time 11 = 25.84 
Similarly, 

Diff_stock_price at time 12 = (Stockprice at time 12) - (Stockprice at time 11) 

0.21 = Stockprice at time 12 - 25.84 
Stock price at time 12 = 25.84 + 0.21 
Stock price at time 12 = 26.05 
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Repeating the same calculations, you get the stock price at time point 13 as 25.94 (Table 12-9). 


Table 12-9. Forecasted Values in the Time Series stock_price and 
diff_stock_price (After Integration) 


Obs 

stock price 

diff stock price 

l 

27.05 


2 

27.23 

0.18 

3 

25.55 

-1.68 

4 

24.86 

-0.69 

5 

24.72 

-0.14 

6 

24.64 

-0.08 

7 

25.5 

0.86 

8 

25.41 

-0.09 

9 

25.34 

-0.07 

10 

25.94 

0.60 

11 

25.84 

-0.10 

12 

26.05 

0.21 

13 

25.94 

-0.11 


The rule of thumb is that you subtract the previous lag values while differentiating; you add previous 
lag values while integrating. Once the series is stationary, it's ready for analysis, and forecasting becomes 
relatively easy. The next step is to identify the type of series. Is it AR, MA, or ARMA? 

For the current example series, you have already identified the order of integration. Now you need to 
identify p and q. This is the next step. 


Step 2: Identifying the Model 

Model identification is one of the important steps in time-series modeling using the Box-Jenkins 
methodology. The final forecast accuracy will depend on how precisely you identify the model. The model 
identification includes finding out the type of time-series process, whether you are dealing with AR(1) or 
AR(2) or MA(1) or MA(2) or ARMA(1,1) or ARMA(2,1), and so on. By identifying the type of series, you will 
get a basic idea of the data. If it is AR, then you know that the current values of the series are dependent on 
the previous values. An AR series has a long-term trend. If it is MA, then the errors will have some impact. 
An MA series has some short-term seasonal effects. In fact, you can write the basic form of the series for 
ARMA(p,q) as follows: 


Y=a*Y, + a,*Y, + a,*Y, 


+.+ a *Y + s, + b *s + b *e 

p t-p t 1 t-1 2 t- 


.+ b *e, 

q t-q 


Identifying the model is definitely not easy. It is not enough to look at the series graph and conclude the 
type and order of the series. You need some concrete metrics for identifying the type and order of a series. 
You need to use the auto correlation function (ACF) and partial auto correlation function (PACF) functions 
and plots to identify the type of the series. I will go into details of this in the following sections. First, for a 
better comprehension, let's look at some time-series equations and their plots. 
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Time-Series Examples 

To understand the identification of the series, you will consider some simulated time series. You will see 
plots for about 15 graphs with their equations (Figures 12-16 through 12-30). The following are some 
time-series graphs. Let's try to guess what their series will be. 

Time Series Plot - 1 (Figure 12-16): y t = 0.66y t ] + s t 



Time Points 


Figure 12-16. A time-series plot for y = 0.66y tl + e t 

Time Series Plot - 2 (Figure 12-17): y t = 0.83y t | + s t 



Time Pomts 


Figure 12-1 7. A time-series plotfor y = 0.83y t t + e t 
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Time Series Plot - 3 (Figure 12-18): y t = -0.95y t | + s t 



Tme Pouts 


Figure 12-18. A time-series plotfor y t = -0.95y t l + e t 

Time Series Plot - 4 (Figure 12-19): y t = 0.4y M + 0.48y t 2 + e t 



Time Ponts 


Figure 12-19. A time-series plotfory t = 0.4y t I + 0.48y t2 + e t 
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Time Series Plot - 5 (Figure 12-20): y t = 0.5y t 1 + 0.4y t2 + s t 



Time Points 


Figure 12-20. A time-series plot for y = 0.5y tI + 0.4y t2 + e t 

Time Series Plot - 6 (Figure 12-21): y t = 0.3y t l + 0.33y t 2 + 0.26y t 3 + s t 



Time Pc*nts 


Figure 12-21. A time-series plot for y = 0.3y tI + 0.33y t2 + 0.26y t3 + e t 
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Time Series Plot - 7 (Figure 12-22): y t = s t -0.8 s t ] 



Time Points 


Figure 12-22. A time-series plotfor y t = e t -0.8 e 

Time Series Plot - 8 (Figure 12-23): y t = s t +0.66 e t ] 



Figure 12-23. A time-series plot for y t = e t +0.66 e t 
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Time Series Plot - 9 (Figure 12-24): y t = s t +0.95 s t 1 



Time Pewits 


Figure 12-24. A time-series plot for y t = e t +0.95 e 

Time Series Plot - 10 (Figure 12-25): y t = e ( -0.4 s t ^0.38 s t ^ 



Time Points 


Figure 12-25. A time-series plot for y t = e t -0.4 e tl -0.38 e 
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Time Series Plot - 11 (Figure 12-26): y t = s t -0.6 e ( 1 -0.3s t2 



Time Points 


Figure 12-26. A time-series plotfory = £-0.6 e -0.3 e 

Time Series Plot - 12 (Figure 12-27): y t = e t -0.55 e M -0.35 s t2 -0.15 s t 3 



Time Points 


Figure 12-27. A time-series plotfory t = e t -0.55 £ t l -0.35 e t2 -0.15 e t3 
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Time Series Plot - 13 (Figure 12-28): y t = -0.95 y t s t -0.8 s t t 


Senes_Vali.es 



Time Pouts 


Figure 12-28. A time-series plot for y= -0.95 y tl + £-0.8 e tl 

Time Series Plot - 14 (Figure 12-29): y t = 0.77 y t l + e, +0.6 e tl 



Time Points 


Figure 12-29. A time-series plotfor y t = 0.77y t ( + £+0.6 e t 
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Time Series Plot - 15 (Figure 12-30): y t = 0.55y ( t + 0.35y t 2 + s ( -0.5 s t ] 



Time_Po«ls 


Figure 12-30. A time-series plotfor y = 0.55y t ] + 0.35y t + e t -0.5 e t 


Was a simple observation enough to judge the type of series? In the time-series plots from Figures 12-16 
through 12-30, some of them are AR processes. There are examples of AR(1), AR(2), AR(3). Some are pure 
MA processes. There are examples of MA(1), MA(2), and MA(3). Some series are ARMA processes. 

In the preceding examples, I simulated the data; hence, I could write the equations. But in practical 
scenarios you will not have the model equations. You will just have the series values and the series graph. 
You know that it’s not easy to classify any of these series as AR, MA, or ARMA just by looking at the series 
values. You need some concrete metrics for identifying the type of series. You need to use ACF and PACF 
functions and plots to identify the type of the series. I discuss these topics in the following sections. The 
following lines are the 15 time-series equations put at one place so that you can compare them: 


TS - 

1: 

y t 

= 

°- 66 y t .i + s t 

TS - 

2: 

y t 

= 

to" 1 

+ 

m 

CO 

O 

TS - 

3: 

y t 

= 

-0.95 y ui + s t 

TS - 

4: 

y t 

= 

0.4 y t .j + 0.48 y t 2 + £ t 

TS - 

5: 

y t 

= 

0.5 y t _j + 0.4 y t _ 2 + s t 

TS - 

6: 

y t 

= 

0.3 y t .j + 0.33 y t _ 2 + 0.26 y, 

TS - 

7: 

y t 

= 

S t -0.8 E t . 1 

TS - 

8: 

y t 

= 

s t +0.66 E tl 

TS - 

9: 

y t 

= 

s t +0.95 s t 1 

TS - 

10: 

y t 

= 

s t . -0.4 s t . 2 -0.38 s t . 2 

TS - 

11: 

y t 

= 

E t . -0.6 E t l -0.3 s t _ 2 

TS - 

12: 

y t 

= 

s t -0.55 £ tl -0.35 s t . 2 -0.15 

TS - 

13: 

y t 

= 

-0.95 y ui + e t -0.8 E t l 

TS - 

14: 

y t 

= 

0.77 y t . 2 + £ t + 0.6 e m 

TS - 

15: 

y t 

= 

0.55 y t . 2 + 0.35 y t _ 2 + E t -0. 
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Auto Correlation Function 

Auto correlation is the correlation between Yt upon Yt-1. Generally you will find correlation between two 
variables, but here you are finding correlation between Y upon previous values of Y. The auto correlation 
function is a function of all such correlations at different lags. The ACF is denoted by ph, where h indicates 
the lag. 

• ACF(O): Correlation at lagO (p 0 ) = Y and Y t =1 

• ACF(l): Correlation at lagl (p 3 ) = Correlation between Y and Y 

• ACF(2): Correlation at lag2 (p ) = Correlation between Y and Y 

• ACF(3): Correlation at lag3 (p 3 ) = Correlation between Y t and Y 3 

The graphs created using auto correlation values are called auto correlation plots. Figure 12-31 shows 
an example of one such plot. On the x-axis you have the lag values, while the y-axis has the auto correlation 
values. The graph might vary based on the type of the series. 



Figure 12-31. An example auto correlation plot 

In AR models, the current values of the series depend on its previous values. You can expect the auto 
correlation to chart a pattern in AR models. 

Partial Auto Correlation Function 

The partial auto correlation function is the partial correlations between Y and its previous values calculated 
at different lags. The partial correlation is the correlation after removing the effect of the other variables. You 
can visualize partial correlation as the correlation coefficient after removing or excluding the effect of the 
other independent variables. You can also treat it is as an exclusive correlation. Let’s see an example. You can 
expect a credit score to be dependent upon the number of existing loans, income, payment dues, length of 
credit history, and number of dependents. Trying to find a simple correlation between the number of loans 
and credit score will give you the association between the two variables, which will also include the effect of 
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all other variables on the credit score. To find a partial correlation, you need to first eliminate the effect of the 
other impacting variables. For all practical purposes, you can envisage the partial correlation coefficient as a 
regression coefficient. 

For the credit score example, here's the equation: 

Credit score = |3 0 + [( (number of loans)+ (S 2 (payment due)+ P 3 (length of credit history) + P 4 (number of 
dependents) 

The partial correlation between the number of loans and the credit score is P 4 , assuming that there is 
no multicolliniarity. p ] is the exclusive impact of the variable number of loans on the credit score. Now you 
can appreciate that a partial coefficient is the quantified association between two variables, which is not 
explained by any of their mutual associations with any other variable. 

In the context of time-series analysis, partial auto correlation is found by regressing the old values of Y 
on the current value. The PACF is denoted by 9 h , where h indicates the lag. 

• PACF(O): Partial correlation at lagO (0 O ) = Y t and Y t = 1 

• PACF(l): Partial correlation at lagl (Oj) = Regression coefficient of Y , when Y t ] is 
regressed up on Y t 

• PACF(2): Partial correlation at lag2 (0 ) = Regression coefficient of Y when Y ] and 
Y are regressed up on Y 

• PACF(3): Partial correlation at lag3 (0 )= Regression coefficient of Y when Y Y 
and Y are regressed up on Y t 

The graphs created using partial auto correlation values are called partial auto correlation plots. 

Figure 12-32 shows an example of one such plot. On the x-axis you have the lag values, while the y- axis 
has the partial auto correlation values. As is the case with ACF, a PACF graph might vary based on the type 
of series. 



In MA models, the current errors in the series depend on its previous error values. You expect the partial 
auto correlation to follow a pattern in MA models. 
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Creating ACF and PACF Plots in SAS 

Using the ACF and PACF plots, you can get an idea about the type of series. This is the second step in the 
time-series analysis process. You have to first make sure that the time series is stationary. You may have to 
differentiate the series at times to make it stationary; a stationary series is ready for analysis. Once the series 
is ready, you want to categorize what type of series it is. Just to recap, to identify the type of series, you need 
to study the ACF and PACF plots generated for the series. What follows next is the code for creating ACF and 
PACF plots in SAS: 

proc arima data=tsl5 plots=all; 
identify var=Series_Values ; 

run; 


You need to use the plots=all option to generate ACF and PACF plots. The SAS output gives four plots. 

• The original series graph 

• ACF plot 

• PACF plot 

• IACF plot 


Note IACF stands for inverse auto correlation. It will be discussed in later sections. 


Rules of Thumb for Identifying the AR Process 

The ACF for the AR process tails off or dies down to zero. The moment you see an ACF plot showing a 
reducing tendency toward zero, you can safely conclude that the process type is AR. Figure 12-33 shows a 
few examples. 
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Figure 12-33. Example ACF plots showing a reducing tendency toward zero 
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If you observe any such behaviors in your series, then definitely there is an AR component in your 
series. It means the current values of the series depend on some previous values of the series. This is one 
of the reasons why the ACF function shows a dampened or slow demolition of correlation toward zero. Y t 
depends on Y , and Y ] depends on Y ; hence, Y indirectly depends on Y . So, the correlation at lag2 is 
slightly less than what it is at lagl. The correlation at lag 3 is slightly less than the correlation at lag 2. Hence, 
the ACF of the AR process shows this behavior of slow decrement toward zero. 

But this is just the first half of the story. What about the order of the AR process? How do you identify 
whether a series is AR(1) or AR(2) or AR(p)? How do you find the value of P? Once you get a confirmation 
from the ACF plot that the process is AR, you need to look at the PACF to know the order of the series. The 
PACF cuts off for an AR process. There will be no significant partial auto correlation values after a few lags. 
The lag number indicates the order of the series. For example, if ACF slowly dampens to zero and the PACF 
function cuts off at lagl, then the order of the AR process is 1. If ACF slowly dampens to zero and the PACF 
function cuts off at lag2, then the order of the AR process is 2. Figure 12-34 shows a few examples of PACF 
plots for an AR process. 





Counting clockwise from the top left, in graph 1, you see that the PACF cuts off at lagl. So, the order of 
that AR process is 1. The partial correlation value at lagO is always 1. In graph 2 (top right), also the PACF 
cuts off at lagl. In graph 3 (in other words, the bottom left), the PACF cuts off at lag2 so the order of that 
AR process is 2. In the last graph (in other words, the right bottom one), the PACF cuts off at lag 3; hence, 
the order of that AR process is 3. The gray band shows the significance. Any lag that is outside the limits 
of that band is significant. The last graph appears as if it is cutting off at lag 4, but the fourth PACF value is 
nonsignificant enough because it is within the limits of the gray band. 
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Here is the justification for the cutoffs in the PACF graph for the AR process. Imagine an AR process of 
order 1; the Y t ] will have maximum impact on Y t . If you remove the previous impacts from Y t | and Y t and 
calculate the correlation, then you will get a high correlation only for lagl, and the rest of the lags will be 
negligible for the AR(1) process. Similarly, for an AR(2) process, the Y , Y will have a maximum impact on 
Y. If you remove all the previous effects from Y , Y , and Y ( , then the correlation will be high for lagl and 
lag2 only. All the remaining correlations, beyond lag 2, will be negligible for an AR(2) process. Please refer to 
Table 12-10 for a summary. 

Table 12-10. The Rules of Thumb to Identify an AR(P) Process 

AR Process Behavior 

ACF Slowly tails off or diminishes to zero. Either reduces in one direction or reduces in a 

sinusoidal (sine wave) passion. 

PACF Cuts off. The cutoff lag indicates the order of the AR process. 


In the simulated list of time series in Figure 12-16 through Figure 12-30, the first six plots belong to the 
AR series. Let’s take a look at their ACF and PACF plots (Figure 12-35 through Figure 12-40). Once again, 

I would like to remind you that for any new series, in an actual business situation, the series equation will 
not be available. This is just an exercise to get the series equation. 

Time Series Plot 1 (Figure 12-16): y t = 0.66y t l + s t 


Trend and Correlation Analysis for Series_Values 



Figure 12-35. ACF and PACF plots for time-series plot 1 (Figure 12-16) 

Note that in Figure 12-35 the ACF is dying down to zero and the PACF cuts off at lagl. TS 1 is an AR 
process of order 1. 
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Time Series Plot 2 (Figure 12-17): y t = O.R3y i + e t 


Trend and Correlation Analysis for Series_Values 
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Figure 12-36. ACF and PACF plots for time-series plot 2 (Figure 12-1 7) 

Note that in Figure 12-36 the ACF is dying down to zero and the PACF cuts off at lag 1. TS 2 is an AR 
process of order 1. 
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Time Series Plot 3 (Figure 12-18): y t = -0.95y M + s t 


Trend and Correlation Analysis for Series_Values 
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Figure 12-37. ACF and PACF plots for time-series plot 3 (Figure 12-18) 

Note that in Figure 12-37 the ACF is dying down to zero and the PACF cuts off at lag 1. TS 3 is an AR 
process of order 1. 
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Time Series Plot 4 (Figure 12-19): y t = 0.4y M + 0.48y t 2 + s t 


Trend and Correlation Analysis for Series_Values 



Figure 12-38. ACF and PACF plots for time-series plot 4 (Figure 12-19) 

Note that in Figure 12-38 the ACF is dying down to zero and the PACF cuts off at lag 2. TS 4 is an AR 
process of order 2. 
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Time Series Plot 5 (Figure 12-20): y t = 0.5y i | + 0.4y t 2 + s t 



Figure 12-39. ACF and PACF plots for time-series plot 5 (Figure 12-20) 

Note that in Figure 12-39 the ACF is dying down to zero and the PACF cuts off at lag 2. TS 5 is an AR 
process of order 2. 


483 















CHAPTER 12 TIME-SERIES ANALYSIS AND FORECASTING 


Time Series Plot 6 (Figure 12-21): y t = 0.3y t _, + 0.33y t 2 + 0.26y t 3 + s t 


Trend and Correlation Analysis for Series_Values 



Figure 12-40. ACF and PACF plots for time-series plot 6 (Figure 12-21 ) 

Note that in Figure 12-40 the ACF is dying down to zero and the PACF cuts off at lag 3. TS 6 is an AR 
process of order 3. 

Rules of Thumb for Identifying the MA Process 

The method of identifying the MA process is almost the same as the AR process. For an MA process, the 
PACF process slowly dies down, and the ACF cuts off. Since, in an MA process, the current value of the 
series has no direct dependency on the past values, the ACF cuts off. The errors of the series depend on the 
previous errors. This is the cause for the PACF to dampen slowly. Table 12-11 lists the rules of thumb for 
identifying an MA process. 


Table 12-11. The Rules of Thumb to Identify MA Process 


MA Process 

Behavior 

ACF 

PACF 

Cuts off. The cutoff lag indicates the order of the MA process. 

Slowly tails off or diminishes to zero. Either reduces in one direction or reduces in a 
sinusoidal or sine wave pattern. 


There is a small issue with the PACF plots. Since you are dealing with errors, which are generally small 
in magnitude, the PACF plot is not clear, and it’s hard to interpret. You can make use of an inverse auto 
correlation function (IACF) plot as a substitute to a PACF plot. 
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Inverse Auto Correlation Function 

The inverse auto correlation function is an auto correlation function wherein you interchange the roles of 
errors and actual series values. The ACF plot for ARMA (2,0) looks the same as an IACF plot for ARMA(0,2). 
You can use IACF as a substitute to PACF in your analysis. For some models, an IACF graph shows a clear 
trend and is hence easy to interpret. You can comprehend IACF as an auto correlation function calculated 
between the errors. Table 12-12 lists the modified and final rules of thumb for identifying an MA process. 


Table 12-12. The Rules of Thumb to Identify MA Process with IACF 


MA process 

Behavior 

ACF 

PACF/IACF 

Cuts off. The cutoff lag indicates the order of the MA process. 

Slowly tails off or diminishes to zero. Either reduces in one direction or reduces in a 
sinusoidal or a sine wave pattern 


Figures 12-41 through 12-46 show some of the MA series from the earlier examples with their ACF, 
PACF, and IACF plots. 

Time Series Plot 7 (Figure 12-22): y t = s t -0.8 s t 1 


Trend and Correlation Analysis for Series_Values 



Figure 12-41. ACF and PACF plots for time-series plot 7 (Figure 12-22) 

Note that in Figure 12-41 the IACF shows a dampening trend toward zero and the ACF cuts off at lagl. 
TS 7 is an MA process of order 1. 


485 



























CHAPTER 12 TIME-SERIES ANALYSIS AND FORECASTING 


Time Series Plot 8 (Figure 12-23): y t = e +0.66 e t | 


Trend and Correlation Analysis for Series_Values 



Figure 12-42. ACF and PACF plots for time-series plot 8 (Figure 12-23) 

Note that in Figure 12-42 the IACF shows a sine wave dampening toward zero and the ACF cuts off 
at lagl. TS 8 is an MA process of orderl. Since the IACF is not clear, you may expect a lower coefficient for 
previous errors. 
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Time Series Plot 9 (Figure 12-24): y t = s t +0.95 £ t | 


Trend and Correlation Analysis for Series_Values 



Figure 12-43. ACF and PACF plots for time-series plot 9 (Figure 12-24) 

Note that in Figure 12-43 the IACF shows a dampening toward zero and the ACF cuts off at lagl. TS 9 is 
an MA process of orderl. 
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Time Series Plot 10 (Figure 12-25): y t = s t -0.4 K t ] -0.38 e t2 
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Figure 12-44. ACF and PACF plots for time-series plot 10 (Figure 12-25) 

Note that in Figure 12-44 the IACF shows a dampening toward zero and the ACF cuts off at lag2. TS 10 is 
an MA process of order2. 
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Time Series Plot 11 (Figure 12-26): y t = s t -0.6 g ( j-0.3 s t ^ 



Figure 12-45. ACF and PACF plots for time-series plot 11 (Figure 12-26) 

Note that in Figure 12-45 the IACF shows a dampening toward zero and the ACF cuts off at lag2. TS 11 is 
an MA process of order2. 
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Time Series Plot 12 (Figure 12-27): y t = s t -0.55 e t j-0.35 s t 2 -0.15 s t 3 
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Figure 12-46. ACF and PACF plots for time-series plot 12 (Figure 12-27) 

Note that in Figure 12-46 the IACF shows a dampening toward zero and the ACF cuts off at lag3. TS 12 is 
an MA process of order 3. 


Rules of Thumb for Identifying the ARMA Process 

When compared with pure AR and pure MA processes, it’s difficult to identify the orders of an ARMA 
process. If the ACF dampens to zero, then it’s an AR process, and the PACF gives the order of the AR process. 
If the IACF dampens to zero, then it's an MA process, and the order is decided by looking at the ACF plot. 
But for an ARMA process, the ACF, PACF, and IACF will all behave the same way. Though you can quickly 
identify that the process is an ARMA process, you will not be able to decide the orders. So, when the series 
is an ARMA process, then you will not be able to identify the orders by looking at the ACF, PACF, and IACF 
graphs. Table 12-13 summarizes the rule of thumb. Figures 12-47 through Figure 12-49 give the example 
plots of ACF, PACF, and IACF for the ARMA processes discussed in earlier sections. 


Table 12-13. The Rule of Thumb to Identify an ARMA Process 


ARMA Process 

Behavior 

ACF/PACF/IACF 

All of them damp down to zero. 
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Time Series Plot 13 (Figure 12-28): y t = -0.95y t 1 + s t -0.8 e ( 1 


Trend and Correlation Analysis for Series_Values 
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Lag 


Figure 12-47. ACF and PACF plots for time-series plot 13 (Figure 12-28) 

Note that in Figure 12-47 the ACF, PACF, and IACF damp down to zero. TS 13 is an ARMA process. 
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Time Series Plot 14 (Figure 12-29): y t = 0.77y i | + s t +0.6 e tl 
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Figure 12-48. ACF and PACF plots for time-series plot 14 (Figure 12-29) 

Note that in Figure 12-48 the ACF, PACF, and IACF damp down to zero. TS 14 is an ARMA process. 
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Time Series Plot 15 (Figure 12-30): y t = 0.55y ( | + 0.35y t 2 + s t -0.5 k ( ] 
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Figure 12-49. ACF and PACF plots for time-series plot 15 (Figure 12-30) 


Note that in Figure 12-49 the ACF, PACF, and IACF damp down to zero. TS 15 is an ARMA process. 


Finding Out ARMA Orders (p,q) Using SCAN and ESACF 

You have already learned that just by looking at the ACF, PACF, and IACF graphs, you will not be able 
to identify the orders p and q in an ARMA process. Fortunately, you have some advanced techniques 
for determining the orders of ARMA models. To determine the orders, you need to calculate two more 
functions: the smallest canonical correlation (SCAN) and extended sample auto correlation function 
(ESACF). They were developed by Ruey Tsay and George Tiao. According to their research, if a series is 
ARMA (p,q), then the ESACF plot is a triangle with (p,q) as the vertex. And the SCAN plot is a rectangle with 
(p,q) as the vertex. The details of this theory are beyond the scope of this book; however, as usual, you can 
use SAS to calculate the ESACF and SCAN function values and arrive at the values of p and q. What follows is 
the SAS code to calculate the ESACF and SCAN function values: 

proc arima data= TS13 plots=all; 
identify var=x SCAN ESACF ; 

run; 


You need to add just two options: SCAN and ESACF. This code gives much more elaborate output. You 
can directly look at the final consolidated recommendations by SCAN and ESACF (Table 12-14). 
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Table 12-14. Finding the Order of an ARMA 
Process Using SCAN and ESACF, SAS Output 


ARMA(p+d,q) Tentative Order Selection Tests 
SCAN ESACF 


p+d 

q 

p+d 

q 

1 

i 

1 

i 

4 

0 

5 

4 

0 

5 

0 

5 


Since you already tested the stationarity well before this phase, you can simply take d as zero here. 

In other words, you can take p+d as p in Table 12-14. The first option suggestion by the SCAN function is 
ARMA(1,1). The ESACF also suggests ARMA(1,1) as the first option. So, the underlined series is 1,3. Both 
SCAN and ESACF give a few more options for p and q. Generally, a less complicated model with fewer 
parameters should be chosen over more complicated ones. Simple models yield better forecasts. 

If you take a model of p=5 and q=5, then the model equation will have too many parameters that need 
to be estimated. The model (p=5,q=5) will have al,a2,a3,a4,a5, and bl,b2,b3,b4,b5 as parameters. Generally, 
models with too many parameters are not robust enough to give accurate forecasts. So, less complicated 
models with p<=2 and q<=2 are preferred. If SCAN and ESACF are suggesting more than one option, then it's 
better to go with the model with the smallest values of p and q. 

Similarly, you can find the SCAN and ESACF for TS 14 and TS 15. Refer to Tables 12-15 and 12-16 for the 
output. 

proc arima data = TS14 plots = all; 
identify var = x SCAN ESACF ; 

run; 


Table 12-15. Finding the Order of an ARMA 
Process Using SCAN and ESACF, SAS Output 


ARMA(p+d,q) Tentative Order Selection Tests 
SCAN ESACF 


p+d 

q 

p+d 

q 

1 

i 

1 

i 

3 

0 

2 

i 

0 

4 

3 

i 



5 

i 



0 

4 
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Table 12-16. Finding the Order of an ARMA 
Process Using SCAN and ESACF, SAS Output 
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ARMA(p+d,q) Tentative Order Selection Tests 
SCAN ESACF 


p+d 

q 

p+d 

q 

2 

i 

2 

1 

1 

2 

1 

2 

4 

0 




Again, the first preference given by both SCAN and ESACF is ARMA(1,1). So, TS 14 is an ARMA(1,1) process. 

proc arima data= TS15 plots=all; 
identify var=x SCAN ESACF ; 

run; 


You already took a stationary series, which required no further differentiation. This means that can be 
taken as d=0. Both SCAN and ESACF are giving the first preference as an ARMA(2,1) process. So, the time 
series TS 15 is an ARMA(2,1) process. 

Let’s recap what has happened up to now. You want to forecast the future values. You need the ARIMA 
technique for model building and forecasting. To build a model using the Box-Jenkins approach, you need 
the series to be stationary. As part of step 1, you tested the stationarity. If the series is not stationary, you make 
it stationary by differentiating it. That is where the value of d in ARIMA (p,d,q) is decided. Then in step 2 you 
identify the orders of p and q. Once you have p and q, you can write a skeleton equation of the model. For 
example, you identify a model as AR(1), and then you can write a skeleton equation as follows: 


Now as a next step you will need to find an estimate of a r Depending on the type of series—AR(p), 
MA(q), or ARMA(p,q)—you need to estimate the number of parameters. If the series is AR(2), you need to 
estimate two parameters. 


Identification SAS Example 

An e-commerce web site wants to track the number of unique visitors per day and has collected data for 
nearly one year. And now the company wants to forecast the number of visitors for the coming week. Before 
forecasting, you need to identify what type of series this is. 

The following is the code for identifying the number of unique visitors (per day). Refer to Table 12-17 
and Figure 12-50 for the output. 

ods graphics on; /* Option to display graphs */ 
proc arima data = web_views plots = all; 
identify var = Visitors stationarity = (DICKEY) ; 

run; 
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Table 12-1 7. Output ofPROC AR1MA on web_views Series 


Name of Variable = Visitors 

Mean of Working Series 

8946.432 

Standard Deviation 

1215.248 

Number of Observations 

340 


Autocorrelation Check for White Noise 


To Lag 

Chi-Square 

DF 

Pr> ChiSq 

Autocorrelations 




6 

379.84 

6 

<.0001 

0.687 

0.542 

0.406 

0.310 

0.204 

0.184 

12 

395.13 

12 

<.0001 

0.159 

0.106 

0.069 

0.039 

0.025 

0.010 

18 

410.19 

18 

<.0001 

0.040 

0.025 

0.064 

0.089 

0.112 

0.122 

24 

421.77 

24 

<.0001 

0.135 

0.089 

0.062 

0.018 

0.031 

0.024 


Augmented Dickey-Fuller Unit Root Tests 

Type 

Lags 

Rho 

Pr< Rho 

Tau 

Pr < Tau 

F 

Pr> F 

Zero Mean 

0 

-2.0950 

0.3195 

-1.07 

0.2567 




l 

-1.2110 

0.4378 

-0.81 

0.3664 




2 

-0.8499 

0.4993 

-0.62 

0.4473 



Single Mean 

0 

-105.309 

0.0001 

-7.86 

<.0001 

30.86 

0.0010 


1 

-80.6368 

0.0016 

-6.30 

<.0001 

19.88 

0.0010 


2 

-84.4002 

0.0016 

-6.12 

<.0001 

18.74 

0.0010 

Trend 

0 

-115.577 

0.0001 

-8.28 

<.0001 

34.28 

0.0010 


1 

-91.2041 

0.0007 

-6.67 

<.0001 

22.25 

0.0010 


2 

-96.9292 

0.0007 

-6.43 

<.0001 

20.75 

0.0010 
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Trend and Correlation Analysis for Visitors 
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Figure 12-50. ACF, PACF, and IACF plots from the output ofPROC AR1MA on web^views series 

From Table 12-17, you can interpret that the process is stationary. The ACF is dampening to zero, so the 
process is an AR process. The PACF cuts off at lagl. A close observation of the PACF graph reveals that lag-2 
is not significant. So, the unique visitors time series is an AR (1) process. The skeleton equation of the time 
series is as follows: 


Y t = a 1 *Y t . 1 + e ,_ 


If you can estimate the value of a , then you can use this model for forecasting the number of unique 
visitors for the coming week, which is the ultimate goal. 


Step 3: Estimating the Parameters 


By now you have the model equation. You have already identified the time series as AR(p) or MA(q) or 
ARMA(p,q). You now have to find the model parameters. This model equation looks similar to the regression 
models discussed in Chapter 10 on multiple linear regressions, fust to recap, refer to the following analogy of 
equations. 

Regression equation: y = /?„ + ftx, + P 2 x 2 + P 3 x 3 + /3 4 x 4 + Aa 5 + fe,.+ P p x P 


Time-series equation: Y t = a 4 *Y t . 4 + a 2 *Y t 2 + a 3 *Y t _ 3 + . 


•+V Y .-r + s < + b i 


% ,+ b *s. 


.+ b *£, 

q t-q 
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Like the way you found the regression coefficients, you now need to use some optimization technique 
to find the parameters of the time-series equation. You can use the maximum likelihood estimator 
technique to guess the model parameters. This step is relatively easy. You just need to use the estimate 
option in the SAS code. You need to specify the model equation by mentioning the order. 

Here is the code for estimating the parameters: 

proc arima data=web_views; 
identify var=Visitors ; 
estimate p=l q=0 method=ML; 

run; 


Earlier you already concluded that the unique visitors time-series is an AR(1) process. Hence, 
p=l and q=0. The method=ML option indicates the maximum likelihood method. 

Table 12-18 is the maximum likelihood table created by this code. 

Table 12-18. Output of PROC ARIMA on web_views Series: Estimating the Parameters 


Maximum Likelihood Estimation 


Parameter 

Estimate 

Standard Error 

t Value 

Approx Pr > |t| 

Lag 

MU 

8942.4 

152.37003 

58.69 

<.0001 

0 

AR1,1 

0.68773 

0.03945 

17.43 

<.0001 

l 


MU is the intercept or the mean of the series. An AR(1,1) estimate (in other words, a,) value is 0.68773. 
You can now write a more complete model equation as follows: 

Y t = (0.68773)*Y t] + s t 

You can use this equation for forecasting future values. In addition to the parameter estimate, SAS 
produces detailed output. The following are some notes on how to interpret some of the unfamiliar yet 
important tables in the output. Refer to Table 12-19. 


Table 12-19. One Important Output Table of PROC ARIMA on 
web_views Series: Estimating the Parameters 


Constant Estimate 

2792.49 

Variance Estimate 

781147.7 

Std Error Estimate 

883.8256 

AIC 

5580.809 

SBC 

5588.467 

Number of Residuals 

340 


The AIC and SBC values help you to compare between two models. Low AIC and SBC value models 
are preferred. A low value of Akaike’s information criteria (AIC) and Schwarz’s Bayesian criterion (SBC) 
indicates that the model with the current estimates is the right fit for the data. This helps when you are trying 
to work with various orders such as AR(1) versus AR(2) versus ARMA(1,1), and so on. Refer to Table 12-20 
and Figure 12-51. An explanation is given after Figure 12-51. 
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Table 12-20. Autocorrelation Check of Residuals, an Output Table ofPROCARIMA on web_uiews Series: 
Estimating the Parameters 


Autocorrelation Check of Residuals 


To Lag 

Chi-Square 

DF 

Pr> ChiSq 

Autocorrelations 





6 

9.97 

5 

0.0761 

-0.088 

0.094 

0.019 

0.070 

-0.075 

0.038 

12 

12.67 

11 

0.3153 

0.063 

-0.002 

0.004 

-0.015 

0.004 

-0.059 

18 

17.03 

17 

0.4521 

0.063 

-0.069 

0.035 

0.016 

0.041 

0.018 

24 

24.50 

23 

0.3765 

0.103 

-0.015 

0.037 

-0.069 

0.032 

-0.051 

30 

28.42 

29 

0.4954 

0.044 

0.078 

-0.045 

-0.022 

0.007 

0.001 

36 

34.61 

35 

0.4869 

0.026 

0.051 

0.110 

0.002 

-0.031 

0.008 

42 

40.64 

41 

0.4865 

0.056 

-0.075 

-0.003 

0.061 

-0.015 

0.053 

48 

50.06 

47 

0.3529 

-0.062 

0.131 

0.029 

0.040 

-0.018 

0.016 


Residual Correlation Diagnostics for Visitors 





Figure 12-51. Residual auto correlation check, output table ofPROC ARIMA on web_views Series; estimating 
the parameters 
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A residual auto correlation check helps you to identify whether there is any information left in the data 
after building the model. The model should be good enough to extract all the hidden patterns in the data. If 
there are still some trends left, then you observe them in residual plots. These residual plots show no trend 
in ACF, PACF, and IACF. You also observed that white noise probabilities (Figure 12-51, bottom right) are 
negligible. 

For example, if you try to fit your model wrongly—in other words, instead of an AR( 1) model, you try to 
fit an ARMA(0,0) model—then the residual plots show the trends (Figure 12-52). For a better understanding, 
you can compare the following code with the previous code. Both are the same except the values of p and q. 

proc arima data=web_views; 
identify var=Visitors j 
estimate p=0 q=0 method=ML; 
run; 


Residual Correlation Diagnostics for Visitors 



Figure 12-52. Residual auto correlation check, choosing a wrong model 

You can’t extract all the information with an AR(0,0) model. You see a trend in the ACF plot for 
residuals. The white noise probability is also significantly high. This indicates that there is some information 
still left in the model. This residual information needs to be extracted. 

Once you are done with the model parameters estimation, you are left only with the forecasting. This is 
the last step in the model building. 
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Step 4: Forecasting Using the Model 

Up to now you have worked really hard to find the type of the model, estimate the parameters, and create the 
final model equation. Once you have the final equation, what remains is to substitute the historical values to 
get the forecasts. SAS is there as usual to help you out. You just need to use a forecast option in the code. 

proc arima data=web_views; 

Identify var=Visitors; 

Estimate p=l q=0 method=ml; 

Forecast lead=7; 

run; 

lead=7 indicates the next seven forecasted values, which, in this case, are the unique number of visitors 
that are likely to visit in the next seven days. This code also appends two new components to the output 
(forecast table and forecast graph). Table 12-21 is the forecast table, and Figure 12-53 charts the forecast 
graph from the SAS output. 


Table 12-21. Next Seven Days Forecasts for the Variable Visitors, SAS Output 


Forecasts for variable Visitors 

Obs 

Forecast 

Std Error 

95% Confidence Limits 

341 

8139.5548 

883.8256 

6407.2885 

9871.8211 

342 

8390.2684 

1072.6622 

6287.8892 

10492.6476 

343 

8562.6905 

1151.2361 

6306.3092 

10819.0718 

344 

8681.2696 

1186.5879 

6355.6000 

11006.9391 

345 

8762.8194 

1202.9463 

6405.0880 

11120.5508 

346 

8818.9033 

1210.6063 

6446.1586 

11191.6480 

347 

8857.4736 

1214.2123 

6477.6612 

11237.2861 
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Forecasts for Visitors 



Figure 12-53. Predicted forecast for the visitors, SAS output 

Data is available until observation 340. The values from 341 to 347 are forecasted and displayed along 
with the 95 percent confidence limits. There is a steady increment expected in the number of visitors to 
the web site. If you see a sharp dip, then it can serve as an alert message. If there is a huge increment in the 
forecasts, then you may need to step up the web site’s IT infrastructure, including the number of servers, 
network bandwidth, and so on. Also, you need to make sure that there are no technical glitches. In the current 
case, everything seems to be normal; the forecasted flow of visitors shows a manageable, steady increase. 

This is a logical end to our time-series analysis and forecasting process using the Box-Jenkins ARIMA 
approach. Every analyst wants to accurately forecast the future values by building the best model for the 
available historical data. This can be definitely achieved by using the Box-Jenkins approach. The following 
are some suggestions while building time-series models: 

• Have sufficient historical data, at least 30 data points. Make sure you don’t run 
into too much history. Only the historical values that will have an impact on future 
forecasts should be considered. 

• Do not forecast too far into the future. With one year of data, forecasting the next two 
years is not a good idea. It may be that 10 percent or fewer data points into the future 
is recommended. 

• Remove outliers before building the model. 
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Case Study: Time-Series Forecasting Using the SAS Example 

Air Voice is a mobile network company that has more than 70 million customers. The customer care center 
of this company handles millions of calls per day. The company wants to increase customer satisfaction by 
resolving most of the customer queries within the agreed-upon service level agreements (SLAs). This can be 
done by better resource planning by forecasting the number of calls every week. The following is a typical 
call volume forecasting exercise. 

The daily call volume data is available for the last 300 days. The volume is in multiples of thousands. 
The following is the procedure to analyze and forecast the future call volume values. 

Step 1: Testing the Stationarity 

The following is the SAS code to test the stationarity. It was already explained in the earlier sections. Refer to 
Table 12-22 for the SAS output relevant to this task. 

proc arima data=call_volume; 

identify var=call_volume stationarity=(DICKEY); 

run; 


Table 12-22. Testing the Stationarity of call_volume Time Series, SAS Output 


Augmented Dickey-Fuller Unit Root Tests 

Type 

Lags 

Rho 

Pr < Rho 

Tau 

Pr < Tau 

F 

Pr> F 

Zero Mean 

0 

-0.3118 

0.6116 

-0.75 

0.3904 




l 

-0.4327 

0.5844 

-0.70 

0.4139 




2 

-0.3451 

0.6040 

-0.74 

0.3964 



Single Mean 

0 

-17.2064 

0.0217 

-3.26 

0.0181 

5.44 

0.0242 


1 

-39.0108 

0.0016 

-4.71 

0.0002 

11.14 

0.0010 


2 

-25.7633 

0.0026 

-3.87 

0.0027 

7.59 

0.0010 

Trend 

0 

-17.4227 

0.1110 

-3.18 

0.0899 

5.31 

0.1108 


1 

-40.4434 

0.0007 

-4.70 

0.0009 

11.22 

0.0010 


2 

-26.2061 

0.0169 

-3.79 

0.0183 

7.48 

0.0191 


The process is stationary on the basis of a single mean P-value in the Augmented Dickey Fuller test. 

Step 2: Identifying the Model Type Using ACF and PACF Plots 

The following code creates the ACF and PACF plots to facilitate the identification of model type. Refer to 
Figure 12-54 for the SAS output relevant to this task. 

ods graphics on; 

proc arima data=call_volume plots=all; 
identify var=call_volume stationarity=(DICKEY); 

run; 
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Trend and Correlation Analysis for Call_volume 
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Figure 12-54. ACF and PACF plots for identification of the model type, SAS output 

The ACF is dampening. The PACF and IACF are also decreasing to zero. This should be an ARMA 
process. You will use the SCAN and ESACF options to find the orders. Refer to the following SAS code and 
relevant output in Table 12-23. 

proc arima data=call_volume plots=all; 
identify var=call_volume SCAN ESACF; 

run; 

Table 12-23. SCAN and ESACF table for 
finding ARMA process orders p and q, SAS output 


ARMA(p+d,q) Tentative Order Selection Tests 

SCAN 

ESACF 

p+d q 

p+d q 

1 1 

1 1 

5 0 



The first option given by both SCAN and ESACF is ARMA(1,1). You will try to estimate the parameters 
a : and b,. 


504 

















CHAPTER 12 TIME-SERIES ANALYSIS AND FORECASTING 


Step 3: Estimating the Parameters 

Refer to the following SAS code and the relevant SAS output in Table 12-24 to find out the model equation 
parameters. 

proc arima data=call_volume plots=all; 
identify var=call_volume SCAN ESACF; 
estimate p=l q=l method=ML; 

run; 


Table 12-24. Estimation of the Parameters, Maximum Likelihood Estimation Table from the SAS Output 


Maximum Likelihood Estimation 

Parameter 

Estimate 

Standard Error 

t Value 

Approx Pr > |t| 

Lag 

MU 

9825.6 

201.76908 

48.70 

<.0001 

0 

MA1,1 

-0.68286 

0.04442 

-15.37 

<.0001 

l 

AR1,1 

0.90783 

0.02424 

37.46 

<.0001 

l 


The AR and MA parameters values are now estimated. The model equation is as follows: 

y t =9825.6 + 0.90783y t:1 + s t -0.68286 s t l 


Step 4: Forecasting 

You can now forecast the next week’s call volumes. The applicable SAS code and its interpretation are left 
as an exercise for you. I have already discussed the process in detail, in the earlier sections, for other cases. 
Refer to Table 12-25 for the relevant portion of the SAS output. 


Table 12-25. Forecasts Data for Call Volume, SAS Output 


Forecasts for variable Call volume 

Obs 

Forecast 

Std Error 

95% Confidence Limits 

301 

9876.9777 

198.7791 

9487.3779 

10266.5776 

302 

9872.2397 

373.4869 

9140.2188 

10604.2607 

303 

9867.9384 

471.0529 

8944.6917 

10791.1852 

304 

9864.0336 

538.3305 

8808.9252 

10919.1420 

305 

9860.4887 

588.0196 

8707.9914 

11012.9860 

306 

9857.2705 

626.0134 

8630.3068 

11084.2342 

307 

9854.3489 

655.6732 

8569.2530 

11139.4449 


As you can observe in Table 12-25, there are no major dips or increments expected over the next week. 
No major changes are required in the resource planning! 
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Checking the Model Accuracy 

Testing the model accuracy is important in any model-building exercise. Residual graphs and AIC and SBC 
values tell you how well you have built the model. To test the forecasting accuracy, it might be a good idea to 
put aside some data for testing purposes and later use it to test the model accuracy. For example, in the call 
volume forecasting data, you have data available for 300 days. Of these 300 days of data, you assume that the 
data is available only for 295 days and use the next 5 days of data for testing the accuracy of forecasts. You 
forecast these 5 days of values using the model. You expect a good model to give the forecasts close to the 
actual values. Refer to Table 12-26. 


Table 12-26. Forecasted Values and Errors in the Previous Call Volume Example 


Day 

Observed 

Forecasted 

Abs Error 

Percent Error 

Squared Deviation 

296 

9345.557 

9407.4053 

61.84803 

0.7% 

3825.179 

297 

9275.306 

9444.8852 

169.5787 

1.8% 

28756.94 

298 

9396.698 

9478.9372 

82.23937 

0.9% 

6763.314 

299 

9217.704 

9509.8748 

292.171 

3.2% 

85363.87 

300 

9533.522 

9537.9828 

4.461153 

0.0% 

19.90189 




122.1 

1.3% 

24945.8 


Ideally you would like to have the error be zero or less than 5 percent. The following are some measures 
of errors or accuracy. In the following text, Y is the actual value and Y is the expected value. 

1. Mean absolute deviation (MAD) 

• Average(absolute deviations) 

. mad = Y’' \ y -W 

/n 

2. Mean absolute percent error(MAPE). This is the most widely used measure. 

• Average(percent deviations) 

100 yn \Y,-Y,\ 

n 


• MAPE =- Y" /v 

n ^ i=1 A 

3. Mean square error (MSE) 

• Average(squared deviations) 

• MSE,z:j r '- f Y 

4. Another related measure is root mean square error, which is RMSE = si MSE. 
In the previous example, here are the values: 

• MAD =122.1 


• MAPE =1.3percent 

• MSE =24,945.8 

• RMSE =157.9 
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It is generally a good practice to keep 5 to 10 percent of the sample data for validation purposes. 
The exact percentage will depend on the size of the data set. 


Conclusion 

In this chapter, you learned what a time-series process is. I discussed one of the most widely used time-series 
techniques, that is, the Box-Ienkins ARIMA technique. I also discussed the four major steps in ARIMA model 
building. The ARIMA models are one of the many types of models. There are sessional ARIMA models; there 
are ARIMAX models, which will use an independent variable X along with the historical values of Y; and 
there are heuristic TSI models. I suggest you research other time-series techniques only after developing 
a solid understanding of the basics of ARIMA model building. The model accuracy-testing methodologies 
discussed at the end of the chapter should be taken seriously. It’s good practice for analysts to furnish the 
error rates along with the forecasted values. 
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CHAPTER 13 


Introducing Big Data Analytics 



By this time you have read 12 chapters on the topic of business analytics and can now appreciate the 
theories and practices that make up this domain. But the learning never ends, especially in a fast-growing 
area like this one. By now, you might already be aware that big data is catching up in popularity very quickly. 

A fast-growing number of organizations, notably online ones with a large amount of visitor traffic, are generating 
an unprecedented amount of data daily, which is almost impossible to handle with classic techniques. 

We discussed this a bit in Chapter 1, and in this last chapter of the book, we expand your understanding 
so that you get a feel of the big data domain and understand what tools and techniques are used by the 
experts working in this field. 

Traditional Data-Handling Tools 

In your day-to-day work, you might use one or more of the following tools: 

• Text editor/Notepad: For modifying small amounts of data 

• Microsoft Excel-. For simple and medium calculations 

• Microsoft Access/SQL: For storing and querying relational data 

• SAS/R: For advanced analytics 

• Tableau/QlikView : For data visualization and reporting 

Have you ever tried to open a 2GB file using Notepad? Have you ever worked with an Excel file of 4GB? 

Is it possible to create a 4GB Excel file in the first place? How much time does it take to query a 10GB 
database? Have you ever performed any analytics operation like regression on a 20GB data set? Can SAS or 
R handle 20GB data with equal efficiency as relatively smaller data sets? We’ll present some examples to give 
you a feel of where we are coming from. 


Walmart Customer Data 

About 245 million active customers and members visit Walmart in 11,000 stores under 71 banners in 
27 countries, which includes its e-commerce web sites (http://corporate.walrriart.com/our-story/). 
Imagine that Walmart wants to know its customers’ buying patterns for the last month. Let’s try to get a 
rough estimate of how much data is generated by Walmart in a month. The following calculations may 
give a conservative estimate: 

(245,000,000 customers) x (2 visits per month) x (20 items bought per visit) = 9,800,000,000 rows 
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There might be as many as 9.8 billion rows in one month. For each row, Walmart must be tracking 
details such as the date stamp, item ID, number of items, item price, net price, and so on. Again, using 
a conservative approach, let’s imagine that there are a minimum of six columns in each such row. What 
will the size of the file be with 9.8 billion rows and 6 columns? Generally speaking, for a CSV or TXT file 
with 10,000 rows and 6 columns per row, the approximate file size may be around 210KB. Based on this, 
the size of a similar file with 9.8 billion rows will be approximately 196GB. And this is just one month’s 
conservative estimate for only the billing transactions. Envisage the data for customers' previous purchases, 
demographics data, product details, store details, and so on. A basic file that needs be considered for any 
customer analytics will be much larger than our estimate of 160GB. 


Facebook Data 

Let’s imagine that Facebook (or a social media network of that size) wants to build an effective online 
advertisement (ad) engine. Depending upon its user profiles, Facebook wants to display the most relevant 
ad on each web page. Or it wants to improve the Facebook feeds. Let’s try to estimate the minimum amount 
of data generated by Facebook in a month. 

Facebook has more than 1 billion users. There are 864 million daily active users, fust for the sake of 
simplicity, let’s treat the user activities such as share, likes, status updates, picture uploads, login and logout, 
and so on, as one activity. Consider the following conservative calculations: 

(864,000,000 customers) x (30 activities per month) = 25,920,000,000 rows 

A file with 25.92 billion rows and 6 columns (for example) may be on the order of 510GB. Facebook may 
need to handle at least this much amount of data if it wants to do any meaningful analytics for its ad engine. 
And this is just a simulated example to give you a feel of the amount of data involved in the process. 

Note that the Walmart and Facebook estimates are conservative as far as the amount of data involved. 
The actual data size involved might be much bigger. Some real-life business scenarios might require you to 
handle petabytes of data. What is a petabyte? Refer to Table 13-1. 


Table 13-1. Units of Data 


Unit Measure 

Conversion 

1GB Gigabyte 

1024MB 

1TB Terabyte 

1024GB 

1PB Petabyte 

1024TB = 1024GB x 1024GB = 1,048,576GB 


As an example, consider that a typical DVD stores about 4.7GB of data. To store 1PB of data, you would 
need 223,101 DVDs. 


Examples of the Growing Size of Data 

The real-time data produced and analyzed by some enterprises is sometimes even beyond our imagination. 
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• NASA stores and processes around 32 petabytes of climate data and related 
simulations. 

• Large Hadron Collider (LHC) at CERN produced 30TB of data in 2012. 

By now, it may be clear that you can’t handle these large volumes of data using traditional tools. Does 
big data really mean a large data set? What is the definition of big data? 

What Is Big Data? 

There is no standard definition for big data. The following are a few definitions that are floating around 
the Internet: 

• Any data set that is difficult to handle with conversional or traditional 
data-processing tools is called big data. 

• Data that is difficult to capture, store, process, search, query, transfer, 
analyze, and visualize is called big data. 

• Big data is a complex, varied, rapidly growing, and huge data set that requires new 
architecture, techniques, and algorithms to analyze it. 

• Big data is a data set that can be analyzed and mined for insights, but because of its 
size, complexity, and growth rate, traditional tools fail to process it. 

The following two sections basically revise what we discussed in Chapter 1. 


The Three Main Components of Big Data 

Big data is not just about the size, though the volume of the data plays a big role in the failure of traditional 
tools while handling it. There are three main components of big data: volume, verity, and velocity. The 
traditional methods and tools have their own limitations in processing the data. The following bullets shed 
some more light on each of these characteristics: 

• Volume: The sheer size of the data, in big data analytics, is so large that it becomes 
almost impossible to process. So far we have mentioned several examples, and to 
add some more examples, transactional data from credit card companies, call data 
from a cell phone company, and billing data from a big retailer like Amazon fall 
under this category. Size is considered the first characteristic of defining big data. 

• Variety: The type of data that we are generating these days is limited not just 
to structured numerical tabular data. Examples are audio files from telecom 
companies, video flies from numerous closed-circuit cameras, and blog data from 
many web sites. All these are nonconventional types of data. Other data in this 
category includes text, numerical data, sequences, time series, multidimensional 
arrays, XML, still images, and social media data. These types of data may come 
pooled with the conventional data. Preparing this type of data for analysis is 
obviously not simple. 
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• Velocity. Data is growing day by day at a rapid pace. In some situations, data keeps 
growing so fast that we do not have enough time to analyze it. Imagine a Formula 
One scenario. We invariably have some sensors placed in each super car. The 
data collected by these sensors is analyzed so as to give some suggestions for 
improvements for the next round. We can’t take, for example, three months to 
transform the data, prepare it for analysis, build a model, and finally validate it so that 
some useful recommendations can be made based on this analysis. All this needs to 
happen well before the next lap starts. This type of analytics even uses real-time data 
streaming from car sensors. Table 13-2 compares big data to conventional data. It’s 
reproduced from Chapter 1 for a ready reference. 


Table 13-2. Big Data vs. Conventional Data 


Big Data 

Normal or Conventional Data 

Huge data sets. 

Data set size in control. 

Unstructured data such as text, video, and audio. 

Normally structured data such as numbers and 
categories, but it can take other forms as well. 

Hard-to-perform queries and analysis. 

Relatively easy-to-perform queries and analysis. 

Needs a new methodology for analysis. 

Data analysis can be achieved by using 
conventional methods. 

Need tools such as Apache's Hadoop, Hive, 

HBase, Pig, Sqoop, and so on, to handle big data. 

Tools such as SQL, SAS, R, and Excel alone may be 
sufficient to handle traditional data. 

Raw transactional data. 

The aggregated, sampled, or filtered data. 

Used for reporting, basic analysis, and 
text mining. Advanced analytics is only in a 
starting stage in big data. 

Used for reporting, advanced analysis, and 
predictive modeling. 

Big data analysis needs both programming skills 
(such as Java) and analytical skills to perform 
analysis (as of today). 

Analytical skills are sufficient for conventional 
data; advanced analysis tools don’t require expert 
programing skills. 

Petabytes/exabytes of data. 

Megabytes/gigabytes of data. 

Millions/billions of accounts. 

Thousands/millions of accounts. 

Billions/trillions of transactions. 

Millions of transactions. 

Generated by big financial institutions, Facebook, 
Google, Amazon, eBay, Walmart, and so on. 

Mostly generated by small enterprises. 


Now you know that big data is complex, huge, and rapidly growing. Also, it’s almost impossible for the 
traditional tools to process this kind of data. Why do we really need to analyze big data? Can’t we simply 
use aggregated data or sample data sets and stick to the traditional techniques and tools? What are the 
applications of big data analytics in real life? The following section answers all these questions. 
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Applications of Big Data Analytics 

Not long ago, it was not a common practice to send soft copies of data. We used to mail all the data in the 
form of hard copies, arranged in files and bundles. With the advent of computers, smartphones, and other 
devices, we are able to store and process huge amounts of data. Now the data is growing in an exponential 
manner. There are already some smart companies that are using this vast amount of available data to their 
benefit. Here are some examples: 

• E-commerce retailers: Online retailers such as Amazon and eBay need to handle 
the millions of customers online who are purchasing thousands of products. They 
analyze petabytes of data every day to recommend the right products, improve 
searches, and improve service quality. Performing big data analytics reveals so 
many customer insights such as customer lifetime value, customer loyalty, product 
popularity, and so on. 

• Online entertainment industry: Sites such as Netflix are using big data analytics 
to understand customer sentiments. They are analyzing big data for movie 
recommendations and even for ticket pricing based on the demand. 

• Automobile industry: Car manufacturers collect data from millions of vehicles 
worldwide using sensors. Because of the data’s sheer size, its variety, and the pace 
with which it accumulates, it falls under the big data category. This data is used in 
understanding the maintenance issues with the models, strong and weak points 
in a particular model, customer driving habits, causes of accidents, suggestions on 
quality improvements, and so on. 

• Telecom industry: In addition to the billing and complaints data, telecom companies 
also maintain call detail record data (CDR) and text messages data. These days a 
smartphone is an integral part of human life. By analyzing CDR data, one can get 
some vital analytics on people’s habits such as sleeping hours, most active hours, 
office hours, driving time, and so on. This data can be utilized by the civic authorities 
in planning and managing cities by setting up the right infrastructure at places. 

Numerous such uses can be found once the analytics models are in place. 

• Online social media industry: One of the most popular users of big data is the social 
media industry. Social networking sites such as Facebook, Google+, Twitter, and 
Linkedln generate petabytes of data every day. They analyze this data for friends’ 
recommendations, advertisement management, feed management, and so on. 

These are just a few examples. In fact, big data analytics is also used in the healthcare industry in drug 
invention. The industry uses simulated data rather than conventional lab experiments. Another use of 
big data is video analytics for self-driving cars. In addition, search engines are using big data in indexing, 
searching, and querying. Big data is used in politics too. Many popular politicians have used big data 
analytics along with focused e-mail campaigns to get an estimate of their chances of winning. So, as you see, 
the uses and possibilities are beyond one’s imagination. 

Now that you are familiar with the big data introductory concepts, we will discuss some more 
applications in the form of the following questions, many of which were considered as far-fetched ideas just 
a few years ago: 

• How about getting some real-time recommendations and discounts (maybe on a cell 
phone) when we drop an item into a cart in a supermarket? 

• How about a city with 100 percent self-driven cars? 

• Can we wear some glasses and get complete information of the objects that we see 
through those glasses? 
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• Can we predict the result and change in the winning probability of a football match 
as it happens? 

• In real time can we control the traffic and better manage it during the peak hours? 

• Can we create a robot that can understand the human language and reply in real time? 

• Can we have a smartphone that realizes our blood pressure levels? Can we have an 
app that rings an alarm at the time of a heart attack? 

• Can we analyze a call center’s historic voice data and automatically resolve customer 
queries instead of an agent physically answering a question? 

All this looks too advanced to be true. But these are all possibilities with big data. Unfortunately, big 
data can’t be processed using traditional tools, old techniques, or typical algorithms; we need an alternative 
solution. 

The Solution for Big Data Problems 

The reason for the failure of traditional data-processing techniques when analyzing big data is apparent. The 
conventional tools take the complete data set as one unit. In that way it would be painful to handle the data 
in the orders of petabytes. Sometimes it’s not even possible to save big data files on one machine. We need 
gigantic supercomputers with multiple processors to handle big data using one machine. A supercomputer 
is not a simple machine; it costs millions of dollars. The solution to the big data challenge is to connect 
multiple computers using computer networks (such as LANs), writing the code in such a way that it is 
separately understood and executed at each computer and the results are finally assembled as one output. 
This technique is called distributed computing. 

Distributed Computing 

Even today, the need of high-computing power is sometimes listed as a major limitation in dealing with big 
data. Distributed computing appears to work well in these situations. It works on the principle of "divide and 
rule.” Instead of considering a huge data set as a single unit, you can it cut into pieces or small data chunks. 
You can then save these relatively smaller pieces of data on an array or network of computing devices. 

Each computer in this network is called a node. Distributing the data onto a cluster setup is the first part of 
distributed computing. The second part is to adapt the code for this kind of cluster of computers. 

The following points summarize the distributed computing approach for dealing with big data: 

1. Modify the overall problem into smaller tasks and execute smaller tasks on an 
array of individual computers. That translates to dividing the data into smaller 
blocks and storing the blocks on individual nodes in the cluster. 

2. Collate the results from all the individual machines and deliver a consolidated 
output. 

Let’s take the same example of Facebook or a similar networking site. In a day there are billions of 
activities on this social networking web site. Say you want to get the number of status updates in a day. 
Imagine that the overall status updates file is 500GB. The following could be the simple SQL code to 
calculate the total count of status updates: 

Select count(*) from feb_status /* feb_status is the status updates data set name */ 
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This code will treat the whole 500GB file as one unit, and it might take an unusually long time. What 
if you take a network of 20 computers and divide this 500GB file into 10,000 pieces of 50MB each? You can 
store these 10,000 individual data chunks on 20 computers. By doing so, you are distributing the one big task 
of computation onto 20 machines. Each computer can locally calculate the number of status updates, and 
then the count data can be totaled to get the desired consolidated number of status updates. All 20 machines 
are working in parallel on independent chunks of data. If the overall computing task takes 20 hours using the 
traditional method and a single machine, it might be natural to expect that with this divide-and-rule policy, 
it will take only an hour or less to complete. In practice, using distributed computing, it will take much less 
than an hour to complete the whole task. 

A supercomputer also could have done the same task efficiently. But, generally speaking, even medium 
and small enterprises can afford a cluster of 20 PCs when compared to the prohibitive costs of one large 
supercomputer. Moreover, 20 PCs can be put to other commercial use when there are no big data files to be 
analyzed. A supercomputer, on the other hand, will make sense only for high-end tasks, which might not be 
available all the time. 

The programing model wherein you divide a big computing task into smaller tasks and collate the 
intermediate results to generate the ultimate result is called the MapReduce programming model. 

What Is MapReduce? 

In distributed computing, the aim is to solve a global problem. You write a program to divide this global 
task into smaller ones and finally assign the pieces to individual machines forming a cluster. This divide 
program is called a map function. Once you are done with the local machine tasks, you then write another 
program to sum up the results from the individual map functions to generate a final consolidated result. This 
consolidation function is termed a reduce function. We discuss each function in the following sections. 


Map Function 

The map function is locally executed on all individual chunks of data. The result of a map function comes 
in the form of a key-value pair. These key-value pairs, from different map functions, carry the intermediate 
results. Generally any individual map function is similar to the overall task. The only difference is the size, 
which is considerably smaller for the map function. The outputs of map functions are used as the inputs of 
the reduce function. 


Reduce Function 

The reduce function takes the intermediate results from the map functions and creates the final output. The 
key-value pairs generated by the map functions is sorted and aggregated in the reduce function. The reduce 
function doesn’t act on the individual pieces of data. In fact, it has no interaction with the original input 
data, which was fed only to the map functions. 

Figure 13-1 shows a reduce diagram that explains the overall MapReduce programming model that is 
used for handling big data. It’s a cost-effective and efficient model of processing big data. 
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Figure 13-1. How MapReduce handles big data problems 

In addition to the MapReduce programming model, the following processes also need your attention: 

• Cluster setup: You set up a cluster by creating a network of individual machines. You 
may need to get the help of a system admin for setting up a cluster. 

• Work assignment: Choose some nodes as slaves and assign the map tasks followed 
by the reduce task. You need to write a separate function to assign the work to 
individual nodes. 

• Task scheduling: Create a master node, which will keep track of all the MapReduce 
tasks, meaning the scheduling. You can’t start with a reduce task. That doesn’t mean 
you need to wait until all the map tasks are completed, though. You may need to 
write some separate code for scheduling the tasks. 

• Load balancing: There will be challenges; say one system in the cluster is slow. 

Some nodes in the cluster may have a good configuration. To achieve the desired 
efficiency, you need to assign the tasks based on the capacity of nodes. This may 
require you to write functions for load balancing as well. 

• Fault tolerance: What if one of the nodes shuts down suddenly? Is there any way you 
can retrieve the data in that node? Maybe it is a good idea to replicate that specific 
load chunk onto a different machine and use it as a backup. So, you may need to take 
care of replication also while distributing the data. 
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It appears that MapReduce is a solution to big data problems, but, as you see, there are so many 
complicated interconnected tasks that make it difficult for an average programmer to try. You need multiple 
skills to solve a big data problem using the MapReduce programming model. It would be great if you have 
a readymade framework that takes care of setting up a cluster, adding and deleting nodes, assigning work, 
scheduling tasks, load balancing, fault tolerance, and so on. Apache Hadoop is the one such popular 
framework. Hadoop is open source, and it’s a windfall for those who want to focus on MapReduce. 

What Is Apache Hadoop? 

The solution for the challenges of several issues of distributed computing using the MapReduce programing 
model is the Hadoop framework. The Hadoop framework is an open source tool, built on the Linux 
operating system. 

Hadoop is a framework. It is a simplified platform to write and execute MapReduce and other big data 
tasks. Hadoop has two major components: the Hadoop distributed file system (HDFS) and MapReduce. 

We go into the details of these two components in the following sections. 

Let’s be clear on a few facts. 

• Hadoop is not a database. 

• Hadoop is not a synonym of big data. 

• Hadoop is not a programing language. 


Hadoop Distributed File System 

In the world of distributed computing and MapReduce, there should not be any abnormally big files. All big 
files will be cut into pieces. The file system in Hadoop is called the Hadoop distributed file system. Every 
file in that system will be 64MB or less. If you transfer any file from outside to HDFS, it will be broken into 
pieces of 64MB. In the case of a cluster, HDFS will cut the files into pieces and distribute them to the different 
nodes within the cluster. You can imagine HDFS as a chopper that cuts the data into small chunks. With it, 
developers do not need to worry about the division and distribution of data. All that you need to do is copy 
the file from the local file system to HDFS. In fact, HDFS will even take care of details such as the data block 
locations. There will be replication of data blocks; if one system goes down, then replicated data blocks will 
be used. Figure 13-2 shows how HDFS distributes the data. 
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HDFS datablocks distribution and replication 


Figure 13-2. How HDFS distributes data 

In Figure 13-2, each block has been replicated three times. By default each block size will be 64MB, and 
it will be replicated three times on the network. 
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MapReduce 

MapReduce is a parallel processing programing model. In partnership with HDFS, MapReduce code blocks 
are executed at local locations. HDFS has all the data blocks and related information. MapReduce has all the 
task-related information. MapReduce and HDFS together process big data almost effortlessly. The Hadoop 
framework takes care of coordination between the MapReduce code and the HDFS data blocks. In Hadoop, 
the MapReduce code blocks need to be written in the Java language. It is the map code for each of the local 
data blocks and the reduce code for consolidation. 

In addition to these two main components, the Hadoop framework plays a key role in the following 
processes: 

• Clusters made easy. Hadoop allows users to quickly set up a cluster network. Hadoop 
makes it extremely easy to add and remove a node without disturbing the rest of the 
network. On the Hadoop framework, it is as easy as adding or deleting an IP address. 

Without Hadoop, setting up a computer cluster and maintaining it for coherent 
processing may not be easy. 

• Storing made easy. Hadoop picks one node as the master node, which is known as a 
name node. Remaining nodes are slave nodes, known as data nodes. The data nodes 
contain the actual data blocks, and the name nodes contain the data about the data 
nodes and data block locations (basically the metadata). Hadoop gives an option to 
choose which machines will be the master and which the slaves. 

• Job scheduling made easy. Hadoop automatically takes care of job orchestration. 

MapReduce will have several map and reduce tasks that need to be executed at the 
data nodes. To manage all the resources, Hadoop has a job tracker and a task tracker. 

The job tracker works along with the master node, and its primary function is to 
allocate the jobs to multiple task trackers. Task trackers run at different data nodes. 

Task trackers present the results and report the status of the jobs to the job tracker. 

• Fault tolerance-. Task trackers regularly send some pulses to the job tracker. It helps 
the job tracker to understand the status at each task tracker, whether it is alive or 
dead. If by any chance the job tracker doesn’t receive heartbeat pulses from any task 
tracker, then it quickly allocates the task to a different task tracker on the replicated 
data block. 

• Load balance-. Hadoop automatically runs some backup jobs. If a node is slow and it 
is slowing down the entire MapReduce process, then the job tracker will assign that 
particular task to other high-speed nodes and thereby reduce the number of tasks 
assigned to the slow machine. 

You can see that many bits and pieces related to big data processing are automatically taken care 
of in Hadoop. All this is handled in a proficient manner. Developers don’t need to worry about cluster 
management, scalability, fault tolerance, job scheduling, load balancing, data distribution, and so on. 
Moreover, Hadoop is free and open source. Refer to Table 13-3 for some useful resources for Hadoop. 
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Table 13-3. Some Resources for Hadoop 


Hadoop logo 


Hadoop home page 
Hadoop documentation 
Hadoop download page 


http://hadoop.apache.org/ 

http://hadoop.apache.org/docs/current/ 

http://hadoop.apache.org/releases.html 



The Hadoop framework makes several tasks easy. It automatically takes care of many key components 
in distributed computing. Writing, debugging, and maintaining key code components can be tough even 
for an expert programmer. The core MapReduce programming and other interactions with Hadoop are in 
the Java programing language; you need to be reasonably comfortable with Java programming to effectively 
work with Hadoop. The logic of the code algorithms need to be adjusted for the MapReduce programing 
model. The usual Java code that produces a result in a traditional way might not be the same with HDFS 
and MapReduce. Wouldn’t it be great if some tools were available that could automatically convert your 
conventional SQL scripts or commands into what is required for MapReduce? Fortunately, some tools 
available in the Hadoop ecosystem can help you to simplify MapReduce programing. Apache’s Hive, Pig, 
Sqoop, and Mahout are a few important ones. The following sections have the details. 


Apache Hive 

Hive is data warehouse software. You can install it on the top of Hadoop. Hive automatically converts 
conventional SQL-like queries to MapReduce code. The queries are written in the Hive query language 
(HiveQL). HiveQL is similar to SQL, and it is extremely useful for data analysts for producing some basic 
business intelligence (BI) reports on big data. Hive can’t work independently without HDFS and MapReduce. 
Hive was originally developed by Faceboolc and later made public. Many data analysts use Hive for data 
summarization, reporting, and analytics. Refer to Table 13-4 for some basic resources for Hive. 

Table 13-4. Some Resources for Hive 



Hive home page http: //hive.apache.org/ 

Hive documentation https://cwiki.apache.org/confluence/display/Hive/LanguageManual 

Hive download page http: //hive.apache.org/downloads. html 
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Apache Pig 

Pig is software that is mainly used for extract, transform, and load (ETL) operations. Pig provides an engine 
for executing data flows in parallel. Like Hive, Pig also runs on the top of Hadoop. You need to write Pig Latin 
scripts to interact with the Pig tool. Pig Latin scripts are much easier than Java MapReduce code. Pig Latin 
scripts can be written for data-processing operations such as join, sort, filter, load, transform, and so on. Pig 
is less of a data analytics tool; its major use is for data handling and manipulation. Pig Latin can be extended 
to use customized user-defined functions. Pig was originally developed by Yahoo, and later it was made 
public. Like the way all Hive queries are automatically converted to MapReduce code before execution, Pig 
Latin scripts are also internally converted to MapReduce. The job of writing MapReduce code blocks for ETL 
is considerably simplified by Pig. Refer to Table 13-5 for some basic resources for Pig. 


Table 1 3-5. Resources for Pig 


Pig logo 



Pig home page 
Pig documentation 
Pig download page 


https://pig.apache.org/ 

http://pig.apache. 0 rg/docs/rO.i 2 .l/index.html 

http://pig.apache.org/ 


There are several other utility tools to help big data developers. Interacting with Hadoop, writing 
MapReduce code, and performing analytics on big data are all getting easier every day with these tools. 
The following are a few more useful tools in the Hadoop ecosystem. 


Other Tools in the Hadoop Ecosystem 

The Hadoop ecosystem is like an app store on a smartphone. There is a tool in the Hadoop ecosystem for 
almost every need related to big data handling. Refer to Table 13-6 for more details. 
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Table 13-6. Other Tools in the Hadoop Ecosystem 

Tool Logo Usage 


Apache Sqoop 


OQOCp 


Apache Flume 



Used for efficient transferring of bulk 
data between Hadoop and structured 
data stores such as a relational database 
management system (RDBMS). An 
example is a data transfer between Hadoop 
and MySQL. 

Transferring large amounts of log data, that 
is, streaming data from Hadoop. Flume is 
used for loading real-time streaming data. 


Apache Avro 


Apache Mahout 



A serialization system. It is used for 
modeling and serializing. 



Advanced analytics on big data. Mahout 
gives access to some predictive modeling 
and machine learning algorithms. 


Apache ZooKeeper 



Coordination service. ZooKeeper is used 
for managing large clusters. 


Apache Oozie 



A workflow scheduler system to manage 
Hadoop jobs. 


Apache HBase H P n C H E Provides a fault-tolerant way of storing 

large quantities of sparse data. It is a 
NoSQL database, that is, a database stored 
in nontabular format for Hadoop. 
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Figure 13-3 is a simplified version of the Fiadoop core ecosystem. 
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Figure 13-3. The Hadoop ecosystem 


Companies That Use Hadoop 

Hadoop is a widely used tool. The following are some major companies using Hadoop: 

• Facebook 

• Google 

• IBM 

• Yahoo 

• Amazon 

• AOL 

• Fox Interactive Media 

• New York Times 
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• Adobe 

• eBay 

• Linkedln 

• Ning 

The more complete list is available at http://wiki.apache.org/hadoop/PoweredBy. 

Up to now, we have discussed some important concepts and tools about big data. As we have done 
throughout this book, we’ll give an example of big data analytics so you can see big data analytics in action. 

Big Data Analytics Example 

In this example, we will use data from the Stack Overflow web site (http://stackoverflow.com/). Stack 
Overflow is a privately held web site that was created in 2008 by Jeff Atwood and Joel Spolsky. The web site 
serves as a platform for users to ask and answer questions on a wide range of computer programming topics. 
If you have a specific question about any programing language, you can log in to Stack Overflow and submit 
your question. You can even tag your questions using keywords such as C++, Java, database, SQL, PHP, and so 
on. Stack Overflow has a portal where it shares the data. Enthusiastic data scientists can log in and download 
the data from this portal. We have downloaded a data set from this web site for this big data example. 

This example is to understand various concepts related to big data. In this example, we are using a 
7GB data set. It’s too small a size for big data, but practical considerations limit us to this size in this example. 
The following are the steps to be followed in this example: 

1. Examining the business problem 

2. Getting the data set 

3. Starting Eladoop 

4. Looking at the Hadoop components 

5. Moving data from the local system to Hadoop 

6. Viewing the data on HDFS 

7. Starting Hive 

8. Creating a table using Hive 

9. Executing a program using Hive 

10. Viewing the MapReduce status 

11. Seeing the final result 

Let’s start with the business problem. 


Examining the Business Problem 

The business wants to extract some basic descriptive statistics such as the total number of questions and the 
top ten topics from the given file. Data such as the total number of unique users, the top ten users, and the 
top ten users’ questions is also desired. You could even go to the extent of building a model that detects and 
suggests the tags automatically when a user enters any question on Stack Overflow web site. For the demo 
purposes, we only try to find the total number of questions mentioned in the input file. 
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Getting the Data Set 

The data set comes from Stack Overflow. The size of the data set is 7GB, and it's a text file. This input file 
needs to be analyzed for finding the basic descriptive statistics. 

Starting Hadoop 

Hadoop is installed on a Linux machine. In this example, we are using a single-node, or a pseudocluster, 
Hadoop setup. Basically we are working on one machine only. Since the operating system is Linux, all the 
operations will be in the form of Linux commands executed in a terminal. The first step is to open a terminal 
in Linux. The username of our Linux machine is Hadoop. If the username were neo, then you would have 
neo@local host home. Refer to Figure 13-4. 


hadoop@localhost:/home 


File Edit View Search Terminal 
[hadoop@localhost home]$ | 


Help 


Figure 13-4. Open terminal in Linux 
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To start Hadoop, you need to go to the Hadoop folder. The path of Hadoop may differ from system to 
system. Refer to Figure 13-5. 


hadoop@localhost:~/hadoop-1.0.3 


File Edit View Search Terminal Help 
[hadoop@localhost home]$ cd hadoop/hadoop-l.O. 
[hadoop@localhost hadoop-1.0.3]$ | 


Figure 13-5. The Hadoop folder 

Now, you are inside the Hadoop folder. You can start Hadoop, which will start the different components 
in the ecosystem such as HDFS, the job tracker, the task tracker, and so on. Refer to Figure 13-6. 


B hadoop@local host:-/hadoop-1.0.3 


file Edit ^iew Search Terminal Help 

[hadoop@localhost hadoop-1.0.3]$ bin/start-all.sh (E 

Warning: SHADOOP HOME is deprecated. 

starting namenode, logging to /home/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-h 
adoop-namenode-localhost.localdomain.out 
localhost: Warning: SHADOOP HOME is deprecated, 
localhost: 

localhost: starting datanode, logging to /home/hadoop/hadoop-1.0.3/libexec/../lo 
gs/hadoop-hadoop-datanode-localhost .localdomain.out 
localhost: Warning: SHADOOP HOME is deprecated, 
localhost: 

localhost: starting secondarynamenode, logging to /home/hadoop/hadoop-1.0.3/libe 
xec/../logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out 
starting jobtracker, logging to /home/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-had 
ocalhost.localdomain.out 

localhost: Warning: SHADOOP HOME is deprecated, 
localhost: 

localhost: starting tasktracker, logging to /home/hadoop/hadoop-1.0.3/libexec/.,/logp 
alhost.localdomain.out 
[hadoop@localhost hadoop-1.0.3]S | 


Figure 13-6. Starting Hadoop 
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start-all.shis the shell script that will start Hadoop. If you get a warning that Hadoop is deprecated, 
you can ignore it. The warning will not appear if you are using the latest version of Hadoop. The following 
are some important messages when you start Hadoop. These messages will give the status of the Hadoop 
component’s initiation. 


• starting namenode, logging to /home/hadoop/hadoop-l.0.3/libexec/../logs/ 
hadoop-hadoop-namenode-localhost.localdomain.out 

• starting datanode, logging to /home/hadoop/hadoop-l.0.3/libexec/../logs/ 
hadoop-hadoop-datanode-localhost.localdomain.out 

• startingsecondarynamenode, logging to /home/hadoop/hadoop-l.0.3/ 
libexec/../logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out 

• startingjobtracker, logging to /home/hadoop/hadoop-l.0.3/libexec/../ 
logs/hadoop-hadoop-jobtracker-localhost.localdomain.out 

• localhost: starting tasktracker, logging to /home/hadoop/hadoop-l.0.3/ 
libexec/../logs/hadoop-hadoop-tasktracker-localhost.localdomain.out 


Looking at the Hadoop Components 

You can now observe that all the Hadoop components have started. You can check on what ports the name 
node, data, and secondary name node (the backup for the name node) are running. Refer to Figure 13-7. 
The 3 ps option shows the ports on which these processes (such as the name node, job tracker, task tracker, 
and so on) are running. 


E 


hadoop@localhost:~/hadoop-1.0.3 


File Edit View Search Terminal Help 
[hadoop@localhost hadoop-1.6.3]$ jps 
15126 JobTracker 
14866 DataNode 
14751 NameNode 
14985 SecondaryNameNode 
17446 Jps 
15268 TaskTracker 

[hadoop@localhost hadoop-1.0.3]S | 


Figure 13-7. Jps option to see the processes 


You can also check and manage Hadoop using the Hadoop administration web interface. To do this, 
you need to use a web browser and go to hit p://localhost :50070/dfshealth. jsp. Refer to Figure 13-8. 
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P'*y« - II • fl X 1 

♦ v S # [ B | http://localhost:50070/dfshealth.jsp 
|e Hadoop NameNode localhostlo. J 

Name Node 'local host, localdoma in: 54310' 

Started: Wed Dec 03 10:37:20 PST 2014 

Version: 1.0.3, rl335192 

Compiled: Tue May 8 20:31:25 UTC 2012 by hortonfo 

Upgrades: There are no upgrades in progress. 

Browse the filesystem 
Namenode Logs 


Cluster Summary 

450 files and directories, 172 blocks = 622 total. Heap Size is 31.57 MB / 966.69 MB (3%) 


Configured Capacity 35.18 GB 

DFS Used 37.36 MB 

Non DFS Used : 16.99 GB 

DFS Remaining 18.15 GB 

DFS Used% 0.1 % 

DFS Remaining% : 51.6 % 

Live Nodes : 1 

Dead Nodes : 0 

Decommissioning Nodes : 0 

Number of Under-Replicated Blocks 5 


NameNode Storage: 


Storage Directory 

Type 

State 

/home/hadoop/hadoop-hadoop/dfs/name 

IMAGE_AND_EDITS 

Active 


This is Apache Hadoop release 1.0.3 


Figure 13-8. Hadoop administration web interface 

By clicking the “Browse the filesystem” link in Figure 13-8, you get the file system details shown in 
Figure 13-9. 
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Piay« - II - (fl K * 

♦ v S # | [ o http://localhost,localdomain:50075/t>rowseDirectoryjsp?namenodelnfoPort=50070&dir=/ 
' • HDFS:/ * 

Contents of directory L 


Goto : [/ | | go | 


[Name 

(Type [size Replication Block 

Size Modification Time Permission 

Owner Group 

home dir | 

2013-08-13 04:07 

[rwxr-xr-x 

[hadoop supergroup 

hup 

[dir f 

2013-08-13 04:07 

rwxr-xr-x 

hadoop supergroup 

[user 

[dir f 

2013-08-13 04:03 

[rwxr-xr-x 

[hadoop supergroup 


Go back to DFS home 


Local logs 

Loo directory 

This is Apache Hadoop release 1.0.3 
Figure 13-9. The Hadoop distributed file system 

The following is the cluster summary (Figure 13-8) before you get the data inside Hadoop: 

• 450 files and directories and 171 blocks = 621 total 

• Heap size: 31.57MB / 966.69MB (3 percent) 

• Configured capacity: 35.18GB 

• Distributed file system (DFS) used: 37.36MB 

• Non-DFS used: 21.33GB 

• DFS remaining: 13.81GB 

• DFS used: 0.1 percent 

• DFS remaining: 39.27 percent 

• Live nodes: 1 

• Dead nodes: 0 

• Decommissioning nodes: 0 

• Number of under-replicated blocks: 5 

Moving Data from the Local System to Hadoop 

On the DFS file system, you need to move the data file into the HDFS location. The command is simple. 

bin/hadoop fs -copyFromLocal /home/final_stack_data.txt stack_data 

Type this command and refer to Figure 13-10. This single command will cut the data into many 64MB 
blocks and distribute them on HDFS. 
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[hadoop@localhost hadoop-1.0.3]$ bin/hadoop fs -copyFromLocal /home/hadoop/Desktop/final st 

ack data.txt stack data 

Warning: SHAOOOP HOME is deprecated. 

[hadoop@localhost hadoop-1.0.3]$ | 


Figure 13-10. Copying data from the local file system to the Hadoop distributed file system 


Viewing the Data on HDFS 

Now, as shown in Figure 13-11, the data is successfully loaded on HDFS. 


I I© 1 HDFS:AJser/tiadoop 1 *0* 

Contents of directory /nser/ hadoop 

Goto : |Aiserybadoop ] go 


Go to parent directory 


jName |Type Size 

Replication Block Size Modification Time Permission Owner |Group 

stack data file 6.99 GB 

1 |64 MB 2014-12-03 11:16 [rw-r—r— hadoop supergroup 

[tmp [dir [ 

2013-08-07 04:42 rwxrwxrwx hadoop supergroup 

warehouse dir 

[2013-08-13 04:03 rwxrwxrwx hadoop [supergroup 


Go back to DFS home 


Local logs 

I .op directory 

This is Apache Hadoop release 1.0.3 
Figure 13-11. Data transferred to HDFS 

In Figure 13-11 you can see a new data set named stack data in HDFS. By clicking it, you can see the 
content in that data set. Refer to Figure 13-12. 
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ll'T*. 1:121 10 I 0 74 4 4 II M 2 a Vn t Wk M lilt. 1121 ddldd2ldd4 Vn IM n# IIM. I 121 1 III 2 I I I 10 \n l Wk run IIM. 1121 dOIOdOdOOI Vn IM -» |1 40. 
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Download this flic 

Tall this flic _ 

Chunk *I/p to vie* (In b>tr%, up to file'* DFS block »!/«•): U7to Mink 


Figure 13-12. Snapshot of a data block 

By clicking stack data, you can see the data in the first chunk, as well as information about other data 
blocks. Refer to Figure 13-13. 


|Oj nurD:/usci/nauuup/suicR._u<Kd | nr 

jdjj ...\n ..>ba: nun ii;cnmrv s » v 5^ ...\n . .j, in: nun u:iri, i:icj i j i j 4 a bi 
68 230 222 267 ...\n ..$ NA: Hum [1:76, 1:121 4786 4491 2510 1144 2071 ...\n ..$ NA: num 11:76, 1:121 
[1:76, 1:12) 10 3 8 79 4 4 11 30 2 0 ...\n ..5 NA: num (1:68, 1:12) 0020826804 ...\n ..5 NA: 

1:12] 39 45 2 0 1 4 3 0 13 0 ...\n ..5 NA: num [1:69, 1:12) 04600416 10 1 ...Vn NA: num [1: 

O 8 0 8 8 0 1 3 5 ...\n NA: num [1:66. 1:121 4 3 3 0 8 4 e 0 8 8 .. ,\n .5 NA: num [1:65, 1:12) 0 

1 0 0 16 .. An NA: num [1:69, 1:12] 00060 25 2300 ...Vn ..S NA: num [1:64, 1:12) 2 0 0 0 0 

7 .,,\n ..S NA: num [1:63. 1:12] 32 6 12 5 92 8 29 7 21 20 ...Vn ..$ NA: num 11:63, 1:12) 2 5 12 8 10 
0 6 1 ..An ..$ NA: nun [1:82, 1:12) 152 99 63 57 105 44 28 33 43 49 ..An NA: num [1:81. 1:12) 0 


Download this file 

Tail this file 

Chunk size to view (in bytes, lip to file's DFS block size): 1 32768 Refresh 


Total number of blocks: 112 


-6994704464849978074: 127.0.0.1:50010 
2379884858919099886: 127.0.0.1:50010 
-504502900132832963: 127.0.0.1:50010 


-1196635793087556703: 
-6853924525356089748: 
-6317974525565522553: 
-1501177905694545525: 
-5543086563329533738: 
2116983165683338158: 
4192466691220417216: 
6082014569630813299: 
-3083516310026969882: 


127.0.0.1:50010 

127.0.0.1:50010 

127.0,0.1:50010 

1.27.0,0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0,0.1:50010 


Figure 13-13. Exact location of each data block along with the node IP address 
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CHAPTER 13 INTRODUCING BIG DATA ANALYTICS 


Figure 13-13 shows an important message; because the data set size is 7GB and each block is sized at 
64MB, you get a total number of 112 blocks. Each block address is given. Refer to Figure 13-14. 


-2609655914204062309: 127.0.0.1:50010 
-7005085182424244977: 127.0.0.1:50010 
-6667541116129086501: 127.0.0.1:50010 


5484256018119500820: 127.0.0.1:50010 
8208839047628257524: 127.0.0.1:50010 


2766109055267620894: 

8560439098249530113: 

-8154540198588547564 

4801738609267090161 

9015278876832924082 


127.0.0.1:50010 

127.0.0,1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

1.27.D.J_.ljjR0_lf) 


1085565467444628722: 
3798046273410056334: 
7296897902937293398: 
•8370668727851400401: 
9104204734650985236: 
6311620234711798174: 
6160663678924941510: 
4644802008316034569: 
-8153836824628051043: 


127.0.0.1:50010 

127.0.0.1:50010 

127.J_.QJj5MLQ 

127.0.0.1:50010 

127,0,0.1:50010 

127..Q.0..1:50Q1Q 

127.0,0.1:50010 

127.0.0.1:50010 

127.0,0.1:50010 


8324023128208491095: 

-5038372401501817231: 



325115968952222180: 

-3729272779043017207: 

5644059051038803629: 

8096145564457288508: 

4092415142371904364: 

1919613947396927412: 

-4339549546894785584: 

-8710325722631310689: 

5183222670479419349: 

1152771480347890522: 

-1727689516010445994: 

3216459131957652775: 

1119923936463448805: 

3946101514503684637: 


127.0.0.1:50010 

127.0.0.1:50010 

127.Q.Q, 1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.0.0.1:50010 

127.Q Q.1:5QQ1Q 

127,0.0.1:50010 

127.0.0.1:50010 


G_o_b_a_ck_to_DFS_home 

Local logs 

Lon directory 

This is Apache Hadoop release 1.0.3 


Figure 13-14. Block addresses 
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CHAPTER 13 INTRODUCING BIG DATA ANALYTICS 


Recall that we are not using a multinode cluster. So, you will have only one machine IP address for all 
the blocks of data. You can see all the blocks are stored at the same IP address (Figure 13-14). If the blocks 
were stored on different nodes, then you would see different IP addresses for each block. If you click a 
particular data block, you see its contents. Refer to Figure 13-15. 
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Figure 13-15. Data inside a block 


Now that you have the data on a distributed system, you can get started with analytics. As discussed 
earlier, the task here is to just get some basic descriptive statistics on the data. We used Hive to perform 
our operations. 
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Starting Hive 

To start Hive, you simply need to type the command hive. Refer to Figure 13-16. 


gr] had oop@local host: -/hadoop-1.0.3 


File Edit view Search Terminal Help 
14751 NameNode 
14985 SecondaryNameNode 
17446 Jps 
15268 TaskTracker 

[hadoop@localhost hadoop-1.0.3]$ bin/hadoop fs -copyFromLocal /home/final stack data 
Warning: SHAOOOP HOME is deprecated. 

Usage: java FsShell [-copyFromLocal clocalsrcs ... <dst>) 

[hadoop@localhost hadoop-1.0.3]$ bin/hadoop fs -copyFromLocal /home/hadoop/Desktop/finalst 

ack data.txt stack data 

Warning; SHADOOP HOME is deprecated. 

[hadoop@localhost hadoop-1.0.3)S hive 

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.had 
oop.log.metrics.EventCounter in all the log4j.properties files. 

Logging initialized using configuration in jar:file:/home/hadoop/hive-0.9.0/lib/hive-common- 
0 .9.0.j ar!/hive-log4j.properties 

Hive history file=/tmp/hadoop/hiveJob log hadoop 201412032190 1185944115.txt 
hive> | 




Figure 13-16. Starting Hive 


The show tables command will show the existing tables in Hive. The default and previous table names 
in Hive will appear with this command. Refer to Figure 13-17. 


m 


hadoop@local host:-/had oop-1.0.3 


File Edit View Search Terminal Help 


ordersl 

ordersr 

Time taken: 6.862 seconds 
hives clear 


FAILED: Parse Error: line 1:0 cannot recognize input near 'clear' '<EOFs' '<EOFs' 


hives show tables; 

OK 

customers 

customersf 

customersl 

customersr 

emp 

managed table 

orders 

ordersf 

ordersl 

ordersr 

Time taken: 0.159 seconds 
hives | 


Figure 13-17. Existing tables in Hive 
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Creating a Table Using Hive 

By now you know that Hive is like a SQL engine on top of Hadoop. In the next step, just as you do in SQL, you 
need to design a table and populate it with data. Refer to Figure 13-18 for the code used to create the table. 


hadoop@localhost:~/hadoop-1.0.3 


m 

File Edit view Search Terminal Help 
ordersl 
ordersr 

Time taken: 6.862 seconds 
hive> clear 
> ; 

FAILED: Parse Error: line 1:0 cannot recognize input near 'clear' '<E0F>' '<E0F>' 

hive> show tables: 

OK 

customers 

customersf 

customersl 

customersr 

emp 

managed table 

orders 

ordersf 

ordersl 

ordersr 

|Time taken: 6.159 seconds 

hive> CREATE TABLE stack data table(id BIGINT, title string, body string, tagl string,tag2 
tring,tag3 string,tag4 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' ESCAPED BY 

V ; 

OK 

|Time taken: 0.55 seconds 
hive> I 


Figure 13-18. Creating a table in Hive 


Now that the table is created, the following is the command to populate the table with the data on 
HDFS. Refer to Figure 13-19. 


LOAD DATA INPATH '/user/hadoop/stack_over_data' INTO TABLE stack_data_table; 
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® hadoop@localhost:~/hadoop-1.0.3 


Eile Edit View Search Terminal Help 
OK 

customers 

customersf 

customers! 

customersr 

emp 

managed table 

orders 

ordersf 

ordersl 

ordersr 

stack datatable 

Time taken: 0.364 seconds 

hive> LOAD DATA INPATH '/user/hadoop/stack over data' INTO TABLE stack data table; 

FAILED: Error in semantic analysis: Line 1:17 Invalid path ''/user/hadoop/stack over data'': 
No files matching path hdfs://localhost:54310/user/hadoop/stack over data 
hive> LOAD DATA INPATH '/user/hadoop/stack data' INTO TABLE stackdata table; 

Loading data to table default.stack data table 
OK 

Time taken: 0.79 seconds 
hive> | 


Figure 13-19. Loading the data into a Hive table 


Executing a Program Using Hive 

You have the table in Hive with the data loaded. Now you need to write the HiveSQL code, and it will 
automatically get converted into MapReduce code. Refer to Figure 13-20. Execute the following HiveSQL 
command: 

Select count(*) from stack_data_table ; 
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hadoop@localhost:~/hadoop-1.0.3 


Eile Edit View Search Terminal Help 
stack data table 
Time taken: 0.364 seconds 

hive> LOAD DATA INPATH '/user/hadoop/stack over data' INTO TABLE stack data table; 

FAILED: Error in semantic analysis: Line 1:17 Invalid path *’/user/hadoop/stack over data": 
No files matching path hdfs://localhost:54310/user/hadoop/stack over data 
hive> LOAD DATA INPATH ’/user/hadoop/stack data' INTO TABLE stack data table; 

Loading data to table default.stack data table 
OK 

Time taken: 0.79 seconds 

hive> Select count(•) from stackdatatable ; 

Total MapReduce jobs * 1 
Launching Job 1 out of 1 

Number of reduce tasks determined at compile time: 1 
In order to change the average load for a reducer (in bytes): 

set hive.exec. reducers .bytes. per. reducer=<nuniber> 

In order to limit the maximum number of reducers: 

set hive.exec.reducers.max=<number> 

In order to set a constant number of reducers: 
set mapred.reduce.tasks=<number> 

Starting Job = job 2O1412031037 0901, Tracking URL = http://localhost:50G30/jobdetails.jsp?j 
obid=job 201412031037 0001 

Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job -Draapred.job.tracker=loc 
alhost:54311 -kill job 201412031037 0001 

Hadoop job information for Stage-1: number of mappers: 28; number of reducers: 1 
2014-12-03 21:08:10,875 Stage-1 map = 0%, reduce = 0% 


Figure 13-20. Executing the HiveSQL command 


Viewing the MapReduce Status 

Figure 13-21 shows the status of map and reduce. In this figure, note that the map status is 28 percent and 
the reduce job is completed by 7 percent. 
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hadoop(§localhost:-/hadoop-1.0.3 _ □ 


Eile Edit ¥iew Search Terminal Help 

Starting Job = Job 201412031637 0801, Tracking URL = http://localhost:5O030/)obdctails.jsp’j 
Obld«Job 291412031037 9091 

Kill Coanand = /home/hadoop/hadoop-1.9.3/libexec/../bin/hadoop job Daapred.Job.tracksr=loc 
alhost:54311 -kill job 201412031037 0001 


Hadoop job 

information for Stage-1:’ 

number of mappers: 28: nunber of reducers: 1 

2014-12-03 

21:08:10,875 

Stage-1 

map 

* 

05. 

reduce 


05 




2014-12-03 

21:08:26,012 

Stage-1 

nap 

• 

25. 

reduce 


05 




2014-12-03 

21:08:29,055 

Stage-1 

nap 

M 

45. 

reduce 


05 




2014 12-03 

21:08:34,172 

Stage-1 

nap 

S 

65. 

reduce 


05. emulative 

CPU 

4.93 sec 

2014-12-03 

21:08:35,184 

Stage-1 

nap 

■ 

65, 

reduce 


05, emulative 

CPU 

4.93 sec 

2014 12 03 

21:08:36,194 

Stage-1 

nap 

■ 

65. 

reduce 


05, emulative 

CPU 

4.93 sec 

2014-12-03 

21:08:37,208 

Stage-1 

nap 

S 

75, 

reduce 


05, Cumulative 

CPU 

10.38 

sec 

2014 12-03 

21:08:38,213 

Stage-1 

nap 

m 

75. 

reduce 


05, Cumulative 

CPU 

10.38 

sec 

2014 12-03 

21:08:39,219 

Stage-1 

nap 

M 

75, 

reduce 


05. Cumulative 

CPU 

10.38 

sec 

2014 12-03 

21:08:40,228 

Stage-1 

nap 

* 

75. 

reduce 


05. Cumulative 

CPU 

10.38 

sec 

2014-12-03 

21:08:41,237 

Stage-1 

nap 

s 

75. 

reduce 


05. Cumulative 

CPU 

10.38 

sec 

2014-12-03 

21:08:42,242 

Stage-1 

nap 

■ 

75. 

reduce 


05. emulative 

CPU 

10.38 

sec 

2014-12-03 

21:08:43,258 

Stage-1 

nap 

a 

85, 

reduce 

8 

05. emulative 

CPU 

10.38 

sec 

2014 12-03 

21:08:44,267 

Stage-1 

nap 

a 

85. 

reduce 


05, emulative 

CPU 

10.38 

sec 

2014 12-03 

21:08:45,278 

Stage-1 

nap 

a 

85, 

reduce 

a 

05, emulative 

CPU 

10.38 

sec 

2014-12-03 

21:08:46,355 

Stage-1 

nap 

a 

85, 

reduce 


05, Cumulative 

CPU 

12.75 

sec 

2014-12-03 

21:08:47,364 

Stage-1 

nap 

= 

85. 

reduce 

s 

05, Cumulative 

CPU 

12.75 

sec 

2014 12-03 

21:08:48,373 

Stage-1 

nap 

a 

85. 

reduce 


05, Cumulative 

CPU 

12.75 

sec 

2014-12-03 

21:08:49,398 

Stage-1 

nap 

a 

95. 

reduce 

a 

05. Cumulative 

CPU 

12.75 

sec 

2014 12-63 

21:08:50,404 

Stage-1 

nap 

a 

95. 

reduce 


05, Cumulative 

CPU 

12.75 

sec 

2014 12-63 

21:08:51,408 

Stage-1 

nap 

a 

95. 

reduce 


05, Cumulative 

CPU 

12.75 

sec 

2014 12 03 

21:08:52,412 

Stage-1 

nap 

a 

95, 

reduce 


05, Cumulative 

CPU 

12.75 

sec 

2014 12-03 

■ 

21:08:53,417 

Stage-1 

nap 

a 

95, 

reduce 


05, emulative 

CPU 

12.75 

sec 


261412-03 21:09:16,574 
2014 12-03 21:09:17,578 
2014-12-03 21:09:18,582 
2014 12-03 21:09:19,587 
2014-12-03 21:09:20,593 
2014-12-03 21:09:21.597 
2014-12-03 21:09:22,610 
2014-12-03 21:09:23.616 
2014-12-03 21:09:24.620 
2014-12-03 21:09:25,624 
2014-12-03 21:09:26,637 
2014-12-03 21:09:27,648 
2014-12-03 21:09:28.668 
2014-12-03 21:09:29.685 
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Figure 13-21. The MapReduce intermediate status 










CHAPTER 13 INTRODUCING BIG DATA ANALYTICS 


Once the MapReduce is completed, you get the results. The total time taken to process this data is about 
five minutes. 

You are simply trying to find the total count of rows, which is the business problem chosen for this 
demo. When you execute a HiveSQL command, you see that the map and reduce jobs start automatically. 
The map jobs will be counting rows in each block of data; reduce is for totaling the counts. 


The Final Result 

The final result shows that there are 6,034,195 records (Figure 13-22). Nearly 6 million questions are 
included in the original data input file. 



Figure 13-22. The final count of rows; the completion of the MapReduce process 


Once you understand the basics and see how you got the basic statistics of counting the number of 
rows, you can write the respective HiveQL commands to get other results such as the top ten topics, the 
frequency of each topic, and so on. 


539 





CHAPTER 13 INTRODUCING BIG DATA ANALYTICS 


Conclusion 

In this chapter, you learned how a basic big data problem is handled. Big data analytics is the latest 
buzzword in the market. Big data today is at the beginning stages of its development. A lot of development 
is expected in this field, particularly in the domains of predictive modeling, text mining, video analytics, and 
machine learning. As of now, not every algorithm is converted into a MapReduce program. But in the near 
future, we may have a tool that works on a distributed platform and can perform all the advanced analytics 
tasks for us. This is a rapidly growing field, so you may see several new tools in the near future. For now, as 
an exercise, we suggest you do some research on the other components in the Hadoop ecosystem. 
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Preface 


Coauthor Venlcat Reddy Konasani and I, Shailendra Kadre, both work for Hewlett-Packard in Bangalore, 
India. He and I have been friends for a while, long before he started working for Hewlett-Packard. I had 
been becoming more and more interested in business analytics, using it in my work and thinking about 
doing a book on the subject. I was aware that he was a trainer in business analytics, and I attended a few of 
his training sessions. I was really impressed with his effective, simple presentation style, not to mention the 
content of his lectures. Here, I realized, was the perfect partner for the book I had in mind. It didn’t take long 
for us to decide to join forces to write this book. 

But before starting work on this volume, we spent a couple years looking at books on business analytics 
and predictive modeling. Some of them were really good. But most of them were too intense and deep 
on the theory and mathematics of statistical algorithms, which are an integral part of this subject. Some 
people like books that take that tack, but most practitioners—even those in the industry—don’t have the 
deep background in the math required or the interest in learning it. Working professionals, particularly 
newcomers to the field of business analytics, are not very comfortable with the deep theoretical treatment 
of statistical algorithms generally provided in most of the books available on analytics. The market need 
we discerned, therefore, was to simplify the presentation of algorithms for professionals who don’t need to 
know the details to succeed in their work. Besides, once introduced to the subject, one can always refer to 
the advanced texts on statistics if such academic rigor is required. 

The good news is that today’s analytics software, like SAS, is designed to do most of the math. Thus, we 
strongly felt there was a need for a book like this one, which takes the power of the software into account 
and, at the same time, simplifies the mathematical concepts involved in the process. With this motivation 
in mind, we started our work and strongly feel we have been successful in showing you how to use SAS to 
perform common analytical procedures while providing the basic knowledge of statistics required. The book 
keeps the theoretical part as simple as possible yet uses numerous real business scenarios to explain the 
concepts and the way they are used in the industry. Venkat’s working experience with the world’s leading 
banks and his vast experience working with students as an analytics trainer has come in handy in designing 
the case studies and examples used in this book. 

As usual, readers are the best judge, and we invite you to join us in an active dialogue on this point so 
that subsequent editions can be improved. Please feel free to reach out to us at shailendrakadre@gmail. com 
and 2l.venkat@gmail.com. 

Whenever we look at books on technical subjects, we ask ourselves particular questions to decide 
whether the book is worth the investment of money, time, and effort. We think you probably do the same, so 
here are answers to the questions you may be asking. 

What Is the Authors’ Training and Experience? 

Both of us are working professionals in the field of business analytics with years of experience in real-life 
business problems. We currently work for HP India in Bangalore. We are postgraduates in either engineering 
or statistics from India’s top technology institutes—the Indian Institute of Technology (IIT). This is 
Shailendra’s second book with Apress, after Going Corporate: A Geek's Guide, published in 2011. 
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What Does the Book Cover? 

This book is basically a first course in analytics using SAS. You will get the foundational knowledge 
needed to be successful in the field. It has 13 chapters, each uniquely designed to develop a step-by-step 
understanding of the subject. Numerous real-life examples are provided with data and SAS code so that you, 
after completing this book, feel comfortable working on actual business challenges. Here’s a rundown of the 
content in each chapter: 

• Chapter 1 starts with the basics of business analytics and presents some use cases to 
build a background for the upcoming chapters. We cover some of the most widely 
used analytical techniques. Next we move on to analytical tools including the most 
widely used analytics tool in technology today: SAS. 

• Chapters 2, 3, and 4 impart the basic code-level knowledge you need in order to work 
with the SAS software. In these chapters, we discuss the basics of SAS, followed by 
data handling in SAS, and, finally, important SAS functions and procedures. These 
chapters give you a foundation in the SAS software, which will be required throughout 
this book, while working with real-life business scenarios on business analytics. 

• Chapters 5 and 6 cover the basics of statistics and simple descriptive statistical 
techniques. They are the foundation-building chapters in statistics. 

• Chapter 7 proceeds—since you have a solid foundation on statistics and SAS—to 
the life-cycle steps in analytics. Any analytical problem solving starts with data 
exploration, validation, and data cleaning. In this chapter, we document some of the 
most creative data-cleaning techniques used in the industry. 

• Chapter 8 is all about testing. Testing hypotheses is a concept that is linked to many 
other statistical topics. In this chapter, we explain how to test hypotheses in 
simple terms. 

• Chapter 9 begins the journey into advanced topics. The first steps in predictive 
modeling are correlation and regression. We explain these concepts with abundant 
real-life examples. 

• Chapter 10 shows that regression is a vast subject; it requires more than one chapter 
to explain. We discuss multiple regressions in this chapter, as well as concepts such 
as multicollinearity and adjusted R-square. 

• Chapter 11 explains why logistic regression is a commonly used predictive modeling 
technique. In this chapter, we discuss model building using logistic regression. 

• Chapter 12 demonstrates time-series analysis, along with its applications, as well as 
how to implement the ARIMA technique. 

• Chapter 13 is all about big data. Considering the exponential growth and popularity 
of the big data analytics domain, we felt it was appropriate to end this book with 
the basics of big data. This final chapter introduces big data with real-time data and 
examples that demonstrate the code and use of Hadoop. 
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What Is Not Covered in the Book? 

While this book prepares you well in the basics of analytics and SAS, the following topics are left for you to 
explore in more advanced resources. This list is not an exhaustive one. 

• Advanced SAS programing techniques such as macros 

• Advanced analytical concepts such as optimization, advanced operation research 
techniques, machine learning, and decision trees 

• Advanced big data concepts such as MapReduce coding 

What Are the Unique Features of the Book? 

A simple style of presentation and hands-on, real-life examples are two things we are particularly proud of. 
The following are some unique design features of this book: 

• It contains numerous examples right from the first chapter until the end. 

• Every concept is explained using a business scenario or case study. 

• It offers the right mix of theory and hands-on labs. 

• It is written by industry professionals who are currently working in the field of 
analytics on real-life problems for paying customers. 

• It simplifies complex statistical concepts. 

• It offers SAS code and a sufficient number of data sets. 

• The book is self-sufficient. You don’t need any other resource or reference to grasp 
the concepts. 


Does the Book Contain Hands-on Exercises? 

Yes! Every chapter has demos and hands-on examples. The data sets used in each case study are available to 
you through the book's web page on Apress.com. You can download the data and code files to practice. 


What Industry Examples Does the Book Cover? 

The examples in this book are not limited to any particular domain. We have tried to cover as wide a 
spectrum of industries as possible. The following are some specifics: 

• Analysis of loan defaulter data in the banking domain 

• Examples from retailers such as Walmart 

• Applications of analytics in social media analytics 

• Analysis of call volumes data in a contact center 

• Analysis of smartphone sales predictions 

• Analysis of associations between customer profile variables 

• Forecasting stock prices 
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• Testing machine quality in a manufacturing plant 

• Getting insights into weblog data 

• Predicting risk associated with each customer in loan data 

• And many more 


What Will You Learn in This Book? 

This book will give you a solid, hands-on foundation in business analytics and SAS. You can expect to learn 
the following: 

• How to apply analytics to real-life business problems 

• How to build analytics models and draw useful inferences from them 

• How to write SAS programs and how to handle data in SAS 

• How to perform basic descriptive analysis to gain insights from the data 

• How to explore, validate, and clean data in a systematic way 

• How to find associations between variables 

• How to test an assumption and accept or reject it statistically 

• How to predict a dependent variable using several independent factors 

• How to forecast future values using historical data 

• How to find the probability of winning versus losing 

• How to handle big data 

• And much more 


What Are the Prerequisites for Reading This Book? 

This book doesn’t require a graduate-level degree in math or statistical background. No prior knowledge in 
statistics is in fact required. Studying math at the high-school level should be adequate to grasp most of the 
concepts explained in this book. Basic knowledge of computers and spreadsheets is required, though. An 
elementary knowledge of databases such as Microsoft Access and SQL will also help. A basic knowledge in 
computer programming is a plus. 


Who Is This Book For? 

This book is for working professionals, new graduates, students, and analytics enthusiasts. While designing 
this book, we envisioned our audience as follows: 

• Anybody who wants to get hands-on with analytics 

• Professionals who want to upgrade their analytical skills 

• Anybody who wants to get started with SAS 
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• Graduate and postgraduate statistics and business analytics students 

• Any new college graduate who wants to enter the world of analytics in domains such 
as retail, banking, insurance, and services 

• Lecturers or teachers who want to teach a basic analytics course to students at the 
graduate or postgraduate level 

• Individuals with nonstatistical backgrounds 

• Anyone who wants to get a basic idea of big data analytics 

Can I Use This Book for an Analytics Course? 

You bet! The goal of this book is to impart the concepts rather than just giving information. All the concepts 
covered are self-contained and don’t require any further references. This book can be easily converted 
into a semester’s course as an introduction to SAS, analytics, and predictive modeling. It can be part of any 
undergraduate or postgraduate program that aims to introduce business analytics to students. 

The following is the suggested class plan for a one-semester course. It consists of 30 classroom theory 
and 12 lab sessions. Course instructors can adapt it to suit their specific requirements. 


Chapter 

Contents 

Theory Classes 

Labs 

Chapter 1 

Introduction to Business Analytics and Data Analysis Tools 

2 

0 

Chapter 2 

SAS Introduction 

2 

1 

Chapter 3 

Data Handling Using SAS 

2 

1 

Chapter 4 

Important SAS Functions and Procs 

2 

1 

Chapter 5 

Introduction to Statistical Analysis 

2 

1 

Chapter 6 

Basic Descriptive Statistics and Reporting in SAS 

3 

1 

Chapter 7 

Data Exploration, Validation, and Data Sanitization 

3 

1 

Chapter 8 

Testing a Hypotheses 

3 

1 

Chapter 9 

Correlation and Linear Regression 

3 

1 

Chapter 10 

Multiple Regression Analysis 

2 

1 

Chapter 11 

Logistic Regression 

2 

1 

Chapter 12 

Time-Series Analysis and Forecasting 

2 

1 

Chapter 13 

Introducing Big Data Analytics 

2 

1 


Total 

30 

12 
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